In this blog, we will explore Azure Databricks, a cloud-based analytics platform, and how it can be used to parse a CSV file from Azure storage and then store the data in a database. Additionally, we will also learn how to process stream data and use Databricks notebook in Azure Data Pipeline. Azure Databricks Overview Azure Databricks is an Apache Spark-based analytics platform that provides a collaborative workspace for data scientists, data engineers, and business analysts. It is a cloud-based service that is designed to handle big data and allows users to process data at scale. Databricks also provides tools for data analysis, machine learning, and visualization. With its integration with Azure Storage, Azure Data Factory, and other Azure services, Azure Databricks can be used to build end-to-end data processing pipelines. Parsing CSV File from Azure BlobStorage to Database using Azure Databricks Azure Databricks can be used to parse CSV files from Azure Storage and then store the data in a database. Here are the steps to accomplish this: Configure Various Azure Components 1. Create Azure Resource Group Image 1 2. Create Azure DataBricks Resource Image 2 3. Create SQL Server Resource Image 3 4. Create SQL Database Resource Image 4 5. Create Azure Storage Account Image 5 6. Create Azure DataFactory Resource Image 6 7. Launch Databricks Resource Workspace Image 7 8. Create Computing Cluster Image 8 9. Create New Notebook Image 9 Parsing CSV File from Azure Storage to Database using Azure Databricks Azure Databricks can be used to parse CSV files from Azure Storage and then store the data in a database. Here are the steps to accomplish this: 1. Create a cluster: First, create a cluster in Azure Databricks as above. A cluster is a group of nodes that work together to process data. 2. Import all the necessary models in the databricks notebook %python from datetime import datetime, timedelta from azure.storage.blob import BlobServiceClient, generate_blob_sas, BlobSasPermissions import pandas as pd import pymssql import pyspark.sql Code 1 3. Mount Azure Storage: Next, mount the Azure Storage account in Databricks as follows #Configure Blob Connection storage_account_name = "storage" storage_account_access_key="***********************************" blob_container = "blob-container" Code 2 4. Establish The DataBase Connection #DB connection conn = pymssql.connect(server='****************.database.windows.net', user='*****', password='*****', database='DataBricksDB') cursor = conn.cursor() Code 3 5. Parse CSV file: Once the storage account is mounted, you can parse the CSV file using the following code #get a list of all blob from the container blob_list = [] for blob_i in container_client.list_blobs(): blob_list.append(blob_i.name) # print(blob_list) df_list = [] #Generate SAS key for each file and load to the dataframe for blob_i in blob_list: print(blob_i) sas_i = generate_blob_sas(account_name = storage_account_name, container_name = blob_container, blob_name = blob_i, account_key = storage_account_access_key, permission = BlobSasPermissions(read=True), expiry = datetime.utcnow() + timedelta(hours=12)) sas_url = 'https://' + storage_account_name +'.blob.core.windows.net/' + blob_container + '/' +blob_i print(sas_url) df=pd.read_csv(sas_url) df_list.append(df) Code 4 6. Transform and Store data in a database: Finally, you can store the data in a database using the following code #Truncate Table Sales Truncate_Query = "IF EXISTS (SELECT * FROM sysobjects WHERE name='sales' and xtype='U') truncate table sales" cursor.execute(Truncate_Query) conn.commit() # SQL Query For Table Creation create_table_query = "IF NOT EXISTS (SELECT * FROM sysobjects WHERE name='sales' and xtype='U') CREATE TABLE sales (REGION varchar(max),COUNTRY varchar(max),ITEMTYPE varchar(max),SALESCHANNEL varchar(max),ORDERPRIORITY varchar(max),ORDERDATE varchar(max),ORDERID varchar(max),SHIPDATE varchar(max),UNITSSOLD varchar(max),UNITPRICE varchar(max),UNITCOST varchar(max),TOTALREVENUE varchar(max),TOTALCOST varchar(max),TOTALPROFIT varchar(max))IF NOT EXISTS (SELECT * FROM sysobjects WHERE name='sales' and xtype='U') CREATE TABLE sales (REGION varchar(max),COUNTRY varchar(max),ITEMTYPE varchar(max),SALESCHANNEL varchar(max),ORDERPRIORITY varchar(max),ORDERDATE varchar(max),ORDERID varchar(max),SHIPDATE varchar(max),UNITSSOLD varchar(max),UNITPRICE varchar(max),UNITCOST varchar(max),TOTALREVENUE varchar(max),TOTALCOST varchar(max),TOTALPROFIT varchar(max))" cursor.execute(create_table_query) conn.commit() #Insert Data From Main DataFrame for rows in df_combined.itertuples(index=False,name=None): row = str(list(rows)) row_data = row[1:-1] row_data = row_data.replace("nan","''") row_data = row_data.replace("None","''") insert_query = "insert into sales (REGION,COUNTRY,ITEMTYPE,SALESCHANNEL,ORDERPRIORITY,ORDERDATE,ORDERID,SHIPDATE,UNITSSOLD,UNITPRICE,UNITCOST,TOTALREVENUE,TOTALCOST,TOTALPROFIT) values ("+row_data+")" print(insert_query) cursor.execute(insert_query) conn.commit() Code 5 As, Shown here The data from all the files is loaded to the SQL server Table Image 10 Azure Databricks notebook can be used to process stream data in Azure Data Pipeline. Here are the steps to accomplish this: 1. Create a Databricks notebook: First, create a Databricks notebook in Azure Databricks. A notebook is a web-based interface for working with code and data. 2. Create a job: Next, create a job in Azure Data Factory to execute the notebook. A job is a collection of tasks that can be scheduled and run automatically. 3. Configure the job: In the job settings, specify the Azure Databricks cluster and notebook that you want to use. Also, specify the input and output datasets. 4. Write the code: In the Databricks notebook, write the code to process the stream data. Here is an example code: #from pyspark.sql.functions import window stream_data = spark.readStream \ .format("csv") \ .option("header", "true") \ .schema("<schema>") \ .load("/mnt/<mount-name>/<file-name>.csv") stream_data = stream_data \ .withWatermark("timestamp", "10 minutes") \ .groupBy(window("timestamp", "10 Code 6 How To Use Azure Databrick notebook in Azure Data Factory pipeline and configure the DataFlow Pipeline Using it. Image 11 1. Create ADF Pipeline Image 12 2. Configure Data Pipeline Image 13 3. Add Trigger To the PipeLine Image 14 4. Configure the trigger Image 15 These capabilities make Azure Databricks an ideal platform for building real-time data processing solutions. Overall, Azure Databricks provides a scalable and flexible solution for data processing and analytics, and it's definitely worth exploring if you're working with big data on the Azure platform. With its powerful tools and easy-to-use interface, Azure Databricks is a valuable addition to any data analytics toolkit.
What is Docker? In this article, you will learn to build Docker image from scratch, and deploy and run your application as a Docker container using Dockerfile.Docker allows developers to build, test, and deploy applications quickly and efficiently using isolated and portable containers that run anywhere. How to Create Docker File In order to build the container image, you’ll need to use a Dockerfile. A Dockerfile is simply a text-based file with no file extension. A Dockerfile contains a script of instructions that Docker uses to create a container image.In a Dockerfile Everything on left is INSTRUCTION, and on right is an ARGUMENT to those instructions. Remember that the file name is "Dockerfile" without any extension. To create Docker file from visual studio - Open project folder in visual studio - Right click on project folder - Go to add - Go to docker support - There are two options to build docker file : windows and Linux - Select one of the given option and it will create the docker file Here we have selected windows operating system, so it will create docker file for windows image but if you select Linux operating system then it will create docker file for Linux image and rest of the docker commands will be same for windows as well as Linux How to Build Docker Image We will build our image using the Docker command. The below command will build the image using Dockerfile from the same directory. docker build -t demoimage:1.0 . - t is for tagging the image. - demoimage is the name of the image. - 1.0 is the tag name. If you don’t add any tag, it defaults to the tag named latest. - . means, we are referring to the Dockerfile location as the docker build context. After the image build output will look like below. Now, we can list the images by using this command. docker images Test the Docker Image Now after building the image we will run the Docker image. The command will be, docker run -d -p 4000:80 --name democontainer2 demoimage:1.0 - d flag is for running the container in detached mode. - p flag flag for the port number, the format is local-port:container-port. - --name for the container name, democontainer2 in our case. Docker started our container in the background and printed the Container ID on the terminal. We can check the running container by using the below command. docker ps In a web browser, access http://localhost:4000 and we can see the index page which displays the content in the custom HTML page we added to the docker image. After creating Docker image we can see all the local images in Docker windows Desktop. Go to the images tab and we can see all the images. Go to the containers tab and we can see all the containers also. Here we are using docker desktop for windows,so we have selected switch to windows option,if you are using docker desktop for Linux then select switch to Linux option. Push Docker Image to Docker Hub Docker Hub is a registry service on the cloud that allows you to download Docker images that are built by other communities. You can also upload your own Docker built images to Docker hub.To push our Docker image to the Docker hub, we need to create an account in the Docker hub. After that, execute the below command to log in from the terminal. It will ask for a username and password (if you are login for the first time). Provide the Docker hub credentials. docker login After login, we now need to tag our image with the docker username as shown below. docker tag demoimage:1.0 <username>/<image-name>:tag For example, here hiral1 is the docker hub username. docker tag demoimage:1.0 hiral1/demoimage:1.0 Run docker images command again and check the tagged image will be there. Now we can push our images to the Docker hub using the below command. docker push hiral1/demoimage:1.0 Now we can check this image will be available in our Docker Hub account. We can inspect a container by following command. docker inspect <container-id> We can view Docker logs in a Docker container by following command. docker logs <container-id> And we can stop the running container by following command. docker stop <container-id> Pull and Run Docker Image from Docker Hub To pull image from docker hub use the following command. docker pull <imagename:tag> Docker checks if the image already exists or not, if it is then it does not download further.In our case it is already there.So we need to remove existing image. To remove the docker image use the following command. docker rmi <imagename:tag> Now pull the docker image from docker hub and check for the list of images. To run the pulled image use the below command docker run -d -p 4000:80 --name windowscontainer hiral1/demoimage:1.0 - d flag is for running the container in detached mode. - p flag flag for the port number, the format is local-port:container-port. - --name for the container name, windowscontainer in our case. In a web browser, access http://localhost:4000 and we can see the index page which displays the content in the custom HTML page we added to the docker image. Below is the example of how to pull and run docker image on Linux VM. Here is the output of the docker image on the browser. Rename/Tag Docker Image To rename the docker image tag the image as follows. docker tag <oldimagename:tag> <newimagename:tag> In our case the old image name is hiral1/demoimage , old tag is 1.0 and new image name is newimage/latest.We have not provided a new tag to newimage, so it gives tag - latest by default. Remove Docker Image To remove the docker image use the following command. docker rmi <imagename:tag>
Abstract This article describes a TSQL JSON parser and provides the source. It is also designed to illustrate a number of string manipulation techniques and also eliminate the issues while dealing with the JSON document containing special symbols like (“/” , ”-”....) in T-SQL. With it you can do things like this to extract the data from a JSON file or document which contains noise and complexities. Summary For Implementation The code for the JSON Parser will run in SQL Server 2005, and even in SQL Server 2000 (note: some modifications are necessary). First the function stores all strings in the temporary table, even the name of the elements, since they are 'escapes' in a different way, and may contain, unescaped, brackets, Special Characters which denote objects or lists. These are replaced in the json string by tokens which represent the strings. After this fetch all the json keywords and values for further processing by using the regular expressions, various string functions and a list of SQL queries and variables to store the values for a particular object. And at the last function will return a whole table which contains rows and columns with no noise in the values as the other tables in the particular database. Figure 1:- Json Input Figure 2:- Function Output Background TSQL isn’t really designed for doing complex string parsing which contains special characters and particularly where strings represent nested data structures such as XML, JSON, or XHTML. You can do it but it is not a pretty sight; but If you ever want to do it anyway ? (note You can now do this rather more easily using SQL Server 2016’s built-in JSON support.) But If the SQL Server version is older or not compatible with the built-in JSON support then you can use this customized function to get the desired output by parsing any type of json document. There is so much stuff behind that all happens to you. For example, it could be that DBA doesn’t allow a CLR, or you lack the necessary skills with procedural scripting. Sometimes, there isn’t any application, or you want to run code unobtrusively across databases or servers. The Traditional way for dealing with data like this is to let a separate business layer parse a JSON ‘document’ into some meaningful structure(Like Tree) and then update the database by making a series of calls and lots of sql procedures. This is pretty, but can get more complicated and headache if you need to ensure that the updates to the database are wrapped into one transaction so that if anything goes wrong or any issues occur, then the whole transaction can be rolled back. This is why a TSQL approach has advantages. Adjacency list tables have the same structure whatever the data in them. This means that you can define a single Table-Valued Type and pass data structures around between stored procedures. Converting the data to Hierarchical table form will be different for each application, but is easy with a TSQL. You can, alternatively, convert the hierarchical table into JSON and interrogate that with SQL. JSON format JSON is one of the most popular lightweight markup languages, and is probably the best choice for transfer of object data from a web page. JSON is designed to be as lightweight as possible and so it has only two structures. The first, delimited by curly brackets, is a collection of Key/value pairs, separated by commas. The key is followed by a colon. The first snag for TSQL is that the curly or square brackets are not ‘escaped’ within a string, so that there is no way of partitioning a JSON ‘document’ simply. It is difficult to differentiate a bracket used as the delimiter of an array or structure, and one that is within a string. The second complication is that, unlike YAML, the datatypes of values can’t be explicitly declared. You have to pass them out from applying the rules from the JSON Specification. Implementation The JSON outputter is a great deal simpler, since one can be sure of the input, but essentially it does the reverse process, working from the root of the json document to the leaves. The only complication is working out the indent of the formatted output string. In the implementation, you’ll see a fairly heavy use of PATINDEX.This uses a RegEx. However, it is all we have, and can be pressed into service by chopping the string it is searching (if only it had an optional third parameter like CHARINDEX that specified the index of the start position of the search!). The STUFF function is also important for this sort of string-manipulation work. CREATE FUNCTION [Platform].[parseJSON] (@JSON NVARCHAR(MAX)) RETURNS @hierarchy TABLE ( Element_ID INT IDENTITY(1, 1) NOT NULL /* internal surrogate primary key gives the order of parsing and the list order */ ,SequenceNo [int] NULL /* the place in the sequence for the element */ ,Parent_ID INT NULL /* if the element has a parent then it is in this column. The document is the ultimate parent, so you can get the structure from recursing from the document */ ,[Object_ID] INT NULL /* each list or object has an object id. This ties all elements to a parent. Lists are treated as objects here */ ,[Name] NVARCHAR(2000) NULL /* the Name of the object */ ,StringValue NVARCHAR(MAX) NOT NULL /*the string representation of the value of the element. */ ,ValueType VARCHAR(10) NOT NULL /* the declared type of the value represented as a string in StringValue*/ ) AS BEGIN DECLARE @FirstObject INT --the index of the first open bracket found in the JSON string ,@OpenDelimiter INT --the index of the next open bracket found in the JSON string ,@NextOpenDelimiter INT --the index of subsequent open bracket found in the JSON string ,@NextCloseDelimiter INT --the index of subsequent close bracket found in the JSON string ,@Type NVARCHAR(10) --whether it denotes an object or an array ,@NextCloseDelimiterChar CHAR(1) --either a '}' or a ']' ,@Contents NVARCHAR(MAX) --the unparsed contents of the bracketed expression ,@Start INT --index of the start of the token that you are parsing ,@end INT --index of the end of the token that you are parsing ,@param INT --the parameter at the end of the next Object/Array token ,@EndOfName INT --the index of the start of the parameter at end of Object/Array token ,@token NVARCHAR(200) --either a string or object ,@value NVARCHAR(MAX) -- the value as a string ,@SequenceNo INT -- the sequence number within a list ,@Name NVARCHAR(200) --the Name as a string ,@Parent_ID INT --the next parent ID to allocate ,@lenJSON INT --the current length of the JSON String ,@characters NCHAR(36) --used to convert hex to decimal ,@result BIGINT --the value of the hex symbol being parsed ,@index SMALLINT --used for parsing the hex value ,@Escape INT --the index of the next escape character /* in this temporary table we keep all strings, even the Names of the elements, since they are 'escaped' in a different way, and may contain, unescaped, brackets denoting objects or lists. These are replaced in the JSON string by tokens representing the string */ DECLARE @Strings TABLE ( String_ID INT IDENTITY(1, 1) ,StringValue NVARCHAR(MAX) ) IF ISNULL(@JSON, '') = '' RETURN SELECT @characters = '0123456789abcdefghijklmnopqrstuvwxyz' --initialise the characters to convert hex to ascii ,@SequenceNo = 0 --set the sequence no. to something sensible. ,@Parent_ID = 0; /* firstly we process all strings. This is done because [{} and ] aren't escaped in strings, which complicates an iterative parse. */ WHILE 1 = 1 --forever until there is nothing more to do BEGIN SELECT @start = PATINDEX('%[^a-zA-Z]["]%', @json collate SQL_Latin1_General_CP850_Bin);--next delimited string IF @start = 0 BREAK --no more so drop through the WHILE loop IF SUBSTRING(@json, @start + 1, 1) = '"' BEGIN --Delimited Name SET @start = @Start + 1; SET @end = PATINDEX('%[^\]["]%', RIGHT(@json, LEN(@json + '|') - @start) collate SQL_Latin1_General_CP850_Bin); END IF @end = 0 --either the end or no end delimiter to last string BEGIN -- check if ending with a double slash... SET @end = PATINDEX('%[\][\]["]%', RIGHT(@json, LEN(@json + '|') - @start) collate SQL_Latin1_General_CP850_Bin); IF @end = 0 --we really have reached the end BEGIN BREAK --assume all tokens found END END SELECT @token = SUBSTRING(@json, @start + 1, @end - 1) --now put in the escaped control characters SELECT @token = REPLACE(@token, FromString, ToString) FROM ( SELECT '\b' ,CHAR(08) UNION ALL SELECT '\f' ,CHAR(12) UNION ALL SELECT '\n' ,CHAR(10) UNION ALL SELECT '\r' ,CHAR(13) UNION ALL SELECT '\t' ,CHAR(09) UNION ALL SELECT '\"' ,'"' UNION ALL SELECT '\/' ,'/' ) substitutions(FromString, ToString) SELECT @token = Replace(@token, '\\', '\') SELECT @result = 0 ,@escape = 1 --Begin to take out any hex escape codes WHILE @escape > 0 BEGIN SELECT @index = 0 --find the next hex escape sequence ,@escape = PATINDEX('%\x[0-9a-f][0-9a-f][0-9a-f][0-9a-f]%', @token collate SQL_Latin1_General_CP850_Bin) IF @escape > 0 --if there is one BEGIN WHILE @index < 4 --there are always four digits to a \x sequence BEGIN SELECT --determine its value @result = @result + POWER(16, @index) * (CHARINDEX(SUBSTRING(@token, @escape + 2 + 3 - @index, 1), @characters) - 1) ,@index = @index + 1; END -- and replace the hex sequence by its unicode value SELECT @token = STUFF(@token, @escape, 6, NCHAR(@result)) END END --now store the string away INSERT INTO @Strings (StringValue) SELECT @token -- and replace the string with a token SELECT @JSON = STUFF(@json, @start, @end + 1, '@string' + CONVERT(NCHAR(5), @@identity)) END -- all strings are now removed. Now we find the first leaf. WHILE 1 = 1 --forever until there is nothing more to do BEGIN SELECT @Parent_ID = @Parent_ID + 1 --find the first object or list by looking for the open bracket SELECT @FirstObject = PATINDEX('%[{[[]%', @json collate SQL_Latin1_General_CP850_Bin) --object or array IF @FirstObject = 0 BREAK IF (SUBSTRING(@json, @FirstObject, 1) = '{') SELECT @NextCloseDelimiterChar = '}' ,@type = 'object' ELSE SELECT @NextCloseDelimiterChar = ']' ,@type = 'array' SELECT @OpenDelimiter = @firstObject WHILE 1 = 1 --find the innermost object or list... BEGIN SELECT @lenJSON = LEN(@JSON + '|') - 1 --find the matching close-delimiter proceeding after the open-delimiter SELECT @NextCloseDelimiter = CHARINDEX(@NextCloseDelimiterChar, @json, @OpenDelimiter + 1) --is there an intervening open-delimiter of either type SELECT @NextOpenDelimiter = PATINDEX('%[{[[]%', RIGHT(@json, @lenJSON - @OpenDelimiter) collate SQL_Latin1_General_CP850_Bin) --object IF @NextOpenDelimiter = 0 BREAK SELECT @NextOpenDelimiter = @NextOpenDelimiter + @OpenDelimiter IF @NextCloseDelimiter < @NextOpenDelimiter BREAK IF SUBSTRING(@json, @NextOpenDelimiter, 1) = '{' SELECT @NextCloseDelimiterChar = '}' ,@type = 'object' ELSE SELECT @NextCloseDelimiterChar = ']' ,@type = 'array' SELECT @OpenDelimiter = @NextOpenDelimiter END ---and parse out the list or Name/value pairs SELECT @contents = SUBSTRING(@json, @OpenDelimiter + 1, @NextCloseDelimiter - @OpenDelimiter - 1) SELECT @JSON = STUFF(@json, @OpenDelimiter, @NextCloseDelimiter - @OpenDelimiter + 1, '@' + @type + CONVERT(NCHAR(5), @Parent_ID)) WHILE (PATINDEX('%[A-Za-z0-9@+.e]%', @contents collate SQL_Latin1_General_CP850_Bin)) <> 0 BEGIN IF @Type = 'object' --it will be a 0-n list containing a string followed by a string, number,boolean, or null BEGIN SELECT @SequenceNo = 0 ,@end = CHARINDEX(':', ' ' + @contents) --if there is anything, it will be a string-based Name. SELECT @start = PATINDEX('%[^A-Za-z@][@]%', ' ' + @contents collate SQL_Latin1_General_CP850_Bin) --AAAAAAAA SELECT @token = RTrim(Substring(' ' + @contents, @start + 1, @End - @Start - 1)) ,@endofName = PATINDEX('%[0-9]%', @token collate SQL_Latin1_General_CP850_Bin) ,@param = RIGHT(@token, LEN(@token) - @endofName + 1) SELECT @token = LEFT(@token, @endofName - 1) ,@Contents = RIGHT(' ' + @contents, LEN(' ' + @contents + '|') - @end - 1) SELECT @Name = StringValue FROM @strings WHERE string_id = @param --fetch the Name END ELSE SELECT @Name = NULL ,@SequenceNo = @SequenceNo + 1 SELECT @end = CHARINDEX(',', @contents) -- a string-token, object-token, list-token, number,boolean, or null IF @end = 0 --HR Engineering notation bugfix start IF ISNUMERIC(@contents) = 1 SELECT @end = LEN(@contents) + 1 ELSE --HR Engineering notation bugfix end SELECT @end = PATINDEX('%[A-Za-z0-9@+.e][^A-Za-z0-9@+.e]%', @contents + ' ' collate SQL_Latin1_General_CP850_Bin) + 1 SELECT @start = PATINDEX('%[^A-Za-z0-9@+.e][-A-Za-z0-9@+.e]%', ' ' + @contents collate SQL_Latin1_General_CP850_Bin) --select @start,@end, LEN(@contents+'|'), @contents SELECT @Value = RTRIM(SUBSTRING(@contents, @start, @End - @Start)) ,@Contents = RIGHT(@contents + ' ', LEN(@contents + '|') - @end) IF SUBSTRING(@value, 1, 7) = '@object' INSERT INTO @hierarchy ( [Name] ,SequenceNo ,Parent_ID ,StringValue ,[Object_ID] ,ValueType ) SELECT @Name ,@SequenceNo ,@Parent_ID ,SUBSTRING(@value, 8, 5) ,SUBSTRING(@value, 8, 5) ,'object' ELSE IF SUBSTRING(@value, 1, 6) = '@array' INSERT INTO @hierarchy ( [Name] ,SequenceNo ,Parent_ID ,StringValue ,[Object_ID] ,ValueType ) SELECT @Name ,@SequenceNo ,@Parent_ID ,SUBSTRING(@value, 7, 5) ,SUBSTRING(@value, 7, 5) ,'array' ELSE IF SUBSTRING(@value, 1, 7) = '@string' INSERT INTO @hierarchy ( [Name] ,SequenceNo ,Parent_ID ,StringValue ,ValueType ) SELECT @Name ,@SequenceNo ,@Parent_ID ,StringValue ,'string' FROM @strings WHERE string_id = SUBSTRING(@value, 8, 5) ELSE IF @value IN ('true', 'false') INSERT INTO @hierarchy ( [Name] ,SequenceNo ,Parent_ID ,StringValue ,ValueType ) SELECT @Name ,@SequenceNo ,@Parent_ID ,@value ,'boolean' ELSE IF @value = 'null' INSERT INTO @hierarchy ( [Name] ,SequenceNo ,Parent_ID ,StringValue ,ValueType ) SELECT @Name ,@SequenceNo ,@Parent_ID ,@value ,'null' ELSE IF PATINDEX('%[^0-9-]%', @value collate SQL_Latin1_General_CP850_Bin) > 0 INSERT INTO @hierarchy ( [Name] ,SequenceNo ,Parent_ID ,StringValue ,ValueType ) SELECT @Name ,@SequenceNo ,@Parent_ID ,@value ,'real' ELSE INSERT INTO @hierarchy ( [Name] ,SequenceNo ,Parent_ID ,StringValue ,ValueType ) SELECT @Name ,@SequenceNo ,@Parent_ID ,@value ,'int' IF @Contents = ' ' SELECT @SequenceNo = 0 END END INSERT INTO @hierarchy ( [Name] ,SequenceNo ,Parent_ID ,StringValue ,[Object_ID] ,ValueType ) SELECT '-' ,1 ,NULL ,'' ,@Parent_ID - 1 ,@type RETURN END Code Snippet 1:- ParseJson Function Closure The so-called ‘impedance-mismatch’ between applications and databases is an illusion. if the developer has understood the data correctly then there is less complexity while processing it. But has been trickier with other formats such as JSON. By using techniques like this, it should be possible to liberate the application or website from having to do the mapping from the object model to the relational, and spraying the database with ad-hoc T-SQL that uses the fact/dimension tables or updateable views. If the database can be provided with the JSON, or the Table-Valued parameter, then there is a better chance of maintaining full transactional integrity for the more complex updates. The database developer already has the tools to do the work with XML, but why not the simpler, and more practical JSON? I hope these routines get you started with experimenting with all this for your requirements.
While working on an Embedded Power BI report, I got a requirement to implement Row Level Security. E.g. Region heads can see data of their Region only. This report is accessible from the .Net MVC application. Locations are assigned to users from the application. This report is hosted on Power BI Server and authenticated by a .Net Application with predefined credentials. Sample Data: There are two tables: Finance Users Figure 1:- finance Table Figure 2:- User Multiple Options to achieve the same with Power BI We have found out following options to implement the same: RLS (Row Level Security) Through Query String Control report filters that will use in embedded code There are certain limitations with Option#1 and Option#2 (I will describe them later), so moving on with Option#3 “Control report filters” Implementation with “Control report filters that will use in embedded code” When you embed a Power BI report, you can apply filters automatically during the loading phase, or you can change filters dynamically after the report is loaded. For example, you can create your own custom filter pane and automatically apply those filters to reports to show user-specific insights. You can also create a button that allows users to apply filters to the embedded report. Step 1: First Identify the filter based on your requirement There are five types of filters available Basic - IBasicFilter Advanced - IAdvancedFilter Top N - ITopNFilter Relative date - IRelativeDateFilter Relative time - IRelativeTimeFilter Based on my requirement I have used a Basic filter and you can use others based on your need. If wants to know more about others click here Here in this blog I am continuing with the #1 basic filter method Step 2: Determine the table and column that you wants to filter In my case, I want to filter the table ‘finance’ and my column was ‘country’ because when particular user login the country column must be filtered For example:- user G1 login the country = ‘Germany’ and this column in table ‘finance’ Table:- ‘finance’ AND Column:- ‘country’ Step 3: Put this two things in this code Step 4: Make Dynamic You can see into this code that there is ‘values’ where I pass “USA” and “Canada” that is hardcode but we want that dynamic values that have to be changed based on which user is login That’s why I made variable that contain ‘country’ name which is assigned to particular login user, For example:- the variable name is ‘Array’ If user G1 login this variable return the value [“Germany”] and set this variable to value Like: Note: if login person has multiple locations than that variable should return values like [“Germany”, ”USA”, ”India”] Note: if login person is manger or CEO they can see all the country data for that Variable has to return Null instead of blank array [ ] Step 5: Put This code into embedded code. Step 5.1: Identify your Method User-owns-data( Embed For your Organization) App-owns-data (Embed for your customer) Step 5.2: According to your method put this code in the given place For the #1 method Add this code into [EmbedReport.aspx] file Figure 3: User-owns-data For the #2 method Add this code into [EmbedeReport.cshtml] file Figure 4: App-owns-data Figure 5: Sample And that’s it. Now, try to run the report from the application and it should work as expected. Feel free to reach out to me, if you face any issues. Conclusion I Hope I’ve Shown You How Easy It Is To Implement Row Level Security, So far we have learned step by step process of it. We started with identifying the filter and then finding out the actual table and column that we want to filter on and generating the code. last but not least put this code into our embedded code.
What Is a Docker? Let’s Say you created an application, and that’s working fine in your machine.??????? Figure 1: App Working Fine But in production it doesn’t work properly, developers experience it a lot. Figure 2: Not Working in Production That is when the developer’s famous words are spoken Client: Your application is not working Developer: It works on my machine Figure 3: Client Developer The Reason could be due to: Dependencies Libraries and versions Framework OS Level features Microservices That the developer’s machine has but not there in the production environment. We need a standardized way to package the application with its dependencies and deploy it on any environment. Docker is a tool designed to make it easier to create, deploy, and run applications by using containers. Figure 4: Docker Icon How does docker work? Docker packages an application and all its dependencies in a virtual container that can run on any server. Figure 5: Container Each container runs as an isolated process in the user space and take up less space than regular VMs due to their layered architecture. Figure 6: Architecture So, it will always work the same regardless of its environment. Credit Goes to @codechips
After creating a new Apple ID account, You might find that you can't use it with some Applications, iTunes, or the App Store. Not to worry, this is not a permanent issue; we can resolve it in just a few steps. Usually, when you go to the App Store and enter your Apple ID and Passcode to download any Application, you will get one alert in the PopUp like 'This Apple ID Has Not Yet Been Used in The iTunes Store. Basically, this alert stops you from using the iTunes store or the App Store, and gives you messages like: 1. This Apple ID has not yet been used with the App Store. Please review Your Apple Account Information. 2. This Apple Id has not yet been used with the iTunes Store. Tap Review to Sign in, then review your account information. Most people will follow the above instruction and tap on Review and agree to the Term and Conditions or add the Extra information to their Apple Id Account. But, sometimes it's not worth it. To Resolve this issue Just follow the below Steps: 1. Go to the Setting 2. Open the Profile section 3. Click on Media & Purchase 4. You will show one popup, Just click on the continue 5. Just review your Account Details Conclusion: Here conclusion is that you just need to enable Media & Purchase on your device.
1. To create a login SQL server, Navigate to Security > Logins 2. In the next screen, Enter a. Login Name b. Select SQL Server authentication c. Enter Password for MS SQL create a user with a password You can also create a login using the T-SQL command for SQL server create login and user. CREATE LOGIN MyLogin WITH PASSWORD = MsSQL 3. Give Full Access for Demo Login Login is created If we refresh the Logins, then we can view Login. How To Create a User? You can use any of the following two ways: · Using T-SQL · Using SQL Server Management Studio Providing limited access only to a certain Database You will be creating a user for the Events27_production database. 1. Connect to SQL server to create a new user a. Connect to SQL Server then expand the Databases folder from the Object Explorer. b. Identify the database for which you need to create the user and expand it. c. Expand its Security folder. d. Right-click the Users folder then choose "New User…" 2. Enter User details, you will get the following screen, a. Enter the desired Username b. Enter the Login name (created earlier) User is created for that specific Database. Create User using T-SQL create user <user-name> for login <login-name> create user DemoUser for login Demo Assigning limited permission to a user in SQL Server Permissions refer to the rules that govern the levels of access that users have on the secured SQL Server resources. SQL Server allows you to grant, revoke and deny such permissions. There are two ways to give SQL server user permissions: 1. Connect to your SQL Server instance and expand the folders from the Object Explorer as shown below. Right-click on the name of the user 2. In the next screen, a. Click the Securable option from the left. b. Click on Search 3. In the next window, a. Select "All Objects belonging to the Schema." b. Select Schema name as "dbo" 4. Grant or Revoke permission of a specific table or DB object a. Identify Table you want to Grant Permission b. In Explicit Permission select Grant The user DemoUser is granted SELECT permission on final_backup_tidx_sctionSponsors. Grant Permissions using T-SQL use <database-name> grant <permission-name> on <object-name> to <username\principle> Use Events27_production Go Grant Select on final_backup_tidx_sctionSponsors to DemoUser 5. Providing ROLE to a specific user: a. In the object explorer expand the databases and security folder. b. Expand Roles and right-click on Database Role. c. Click on New database role. Then a new pop-up window is open. d. In the General tab enter the role name and click on ok. 6. Refresh the roles. In below screenshot shows the role Remove Login from SQL Server: 1. To drop login SQL server, Navigate to Security > Logins 2. Select the desired login and click on Delete Drop Login using T-SQL DROP LOGIN Demo;
There has already been plenty said about social media, how it alters consumer behavior, and how businesses may utilize it to turn the tide in their favor. Despite the vast quantity of information available on the internet, many businesses have trouble grasping the concept of "Social Media Marketing" and how it works. As a result, here’s a complete guide that will simplify social media marketing for people who are having trouble coming up with effective marketing ideas. Social media is a worldwide network of over 3.5 billion people that enjoy sharing, seeking, and creating information. Business development tactics have quickly become inextricably linked to social media outlets. When it comes to creating genuine connections with customers, you can't ignore the power of "Social Media". What is Social Media Marketing? It is the process of developing custom content for each social media platform in order to boost engagement and promote your brand. The goal of Social Media Marketing is to engage with your audience or customers and assist them in better understand your company. It is really useful to the success of your company. Just as in the offline world, your Social Media Marketing success depends on the ability to locate and make your target audience happy so that they appreciate your brand and share your stories with others. If your story isn't worth sharing, your Social Media Marketing efforts will be in vain. Importance of Social Media Marketing: When consumers want to learn more about a company or product, they turn to social media since that's where they'll find other people talking about it. If you don't have a presence on social media, you are squandering a fantastic opportunity to create an impression. Every day, numerous times a day, your consumers and prospects use social media networks. Brands wishing to learn more about their customers' interests and preferences could use social media. According to the experts, smart businesses will continue to invest in social media in order to achieve long-term commercial growth. Your business can't afford to ignore social media in this increasingly competitive market. Benefits of Social Media Marketing: You may use a Social Media Marketing strategy to reap a variety of benefits, such as expanding the reach of whatever you're offering by engaging in two-way dialogues with potential clients. The following are the few advantages of Social Media Marketing: 1. Social Media Marketing constantly finds a new audience for your business: Nothing is more intimidating than speaking to a fresh audience or people who have never interacted with your brand. Social Media Marketing gives you access to tools and methods that make it simple to reach out to a new audience. You can utilize content on Facebook and other social media sites to communicate with potential customers. While it can be difficult to capture people's attention, compelling content can help you stand out. 2. Social Media Marketing helps customers form stronger relationships: If you are having a perception that Social Media Marketing is solely for selling and advertising, it’s high time that you should Rethink. To develop long-term relationships, successful brands interact and engage with their social media consumers. Instead of advertising your products or services, you can just ask your social media followers questions about them or offer something that will make their lives easier. You will gain their trust and demonstrate your concern for their wants and opinions in this manner. Before urging people to invest in you, you should always satisfy them first. 3. Social Media Marketing increases Leads and Conversions: Businesses can create leads through platforms like Facebook, Instagram, Twitter, and Linkedin. To increase conversions, you can use a combination of paid and organic methods. To drive potential customers into your sales, there are some of the most effective techniques such as video marketing, paid ad campaigns, giveaways, and email opt-ins. As everything is happening online, Social Media Marketing is a better, faster, and simpler way to establish a database of potential customers. With more visibility, your business will have more conversion opportunities. Your social media followers may be drawn to your business’ website through compelling content, and as a result, they may become your loyal customers. 4. Social Media Marketing gives you an edge over your Competitors: There's a lot to be learned from your competitors' social media presence, especially if you're new to social media and don't have any strong marketing ideas. Progressive businesses keep a close eye on their competitors to evaluate what works and what doesn't. Following what your competitors are doing on social media should be an important element of your social media marketing plan. You might begin experimenting with what your competitors are doing well. 5. Social Media Marketing is economically beneficial: The most cost-effective and diverse way to promote a business is through social media marketing. Most social networking services allow you to create a profile for free. Even the cost of running a paid campaign to promote your content is relatively inexpensive when compared to other advertising platforms. There's always something you can do to increase the performance of your digital marketing. One of the advantages of social media marketing is that you can use real-time data to track your progress and fine-tune your plan. Conclusion - There are more than 200 social networking websites to choose from. Make a name for yourself on at least a couple of them. Be where your target market is. To succeed, keep up with all of the current trends and technologies. There is no better way to build your business than by combining Social Media Marketing with an effective influencer marketing approach.
SSRS or SQL Server Reporting Services is one of the tools available in Microsoft SQL Server Data Tools. It is a server-based reporting platform that you can use to create and manage tabular, matrix, graphical, and free-form reports that contain data from relational and multidimensional data sources. SSRS is a set of readymade tools, that helps you to create, deploy and manage reports. SSRS allows are reports to be exported in various formats (Excel, PDF, word ,CSV ,XML etc) SSRS allows reports to be delivered via emails or dropped to a share location. Advantages of using SSRS Supports a variety of file formats Facility to drill down to different levels of data Helpful and perceptive reporting Access to enterprise-level features Simplistic implementation owing to a centralized server Why SSRS? Here, are prime reasons for using SSRS tool: SSRS is an enhanced tool compared to Crystal Reports Faster processing of reports on both relational and multidimensional data Allows better and more accurate Decision-making mechanism for the users Allows users to interact with information without involving IT professionals It provides a World Wide Web-based connection for deploying reports. Hence, reports can be accessed over the internet SSRS allows reports to be exported in different formats. You can deliver SSRS reports using emails SSRS provides a host of security features, which helps you to control, who can access which report Working and Architecture The main components of SSRS are the following: Report Builder -This component is basically used as a drag and drop utility which can be used to pick any functionality or tables and drag it as per usage. It runs on the client computer. Report Designer - This component is used to develop reports. Complex reports can be developed with ease using this component. It is a publishing tool which is hosted in SSDT (SQL Server Data Tools) or visual studio. Report Manager -To access any web-based reports, we can make use of Report Manager. Report Server - This component is used to store SQL server Engine metadata. Server Database Report - This component is used to store security settings, report definitions, metadata, delivery data, etc. Data Sources - The reporting service components retrieve data from data sources like multidimensional, relational or traditional data sources. Reporting Life Cycle Report Authoring: In this phase, the report author defines the layout and syntax of the data. The tools used in this process are the SQL Server Development Studio and SSRS tool. Management: This phase involves managing a published report which is mostly part of the websites. In this stage, you need to consider access control over report execution. Delivery: In this phase, you need to understand when the reports need to be delivered to the customer base. Delivery can be on-demand or pre-defined schedule. You can also add an automation feature of subscription which creates reports and sends to the customer automatically. What is RDL? Report Definition Language (RDL) is an XML representation of a SQL Server Reporting Services report definition. A report definition contains data retrieval and layout information for a report. Create RDL Report You can create a RDL reports using any of the following reporting tools, Syncfusion Web Report Designer: Provides intuitive user interface to create or edit report online. Microsoft Report Builder: You can create a RDL report using the Microsoft stand-alone Report Builder. Visual Studio Report Server project template: To create a RDL report in Visual Studio, a Report Server project is required where you can save your report definition (.rdl) file. How To create a report server project From the File menu, select New > Project. In the left-most column under Installed, select Reporting Services Select the Report Server Project icon Creating a report definition file (RDL) In the Solution Explorer pane, right-click on the Reports folder. If you don't see the Solution Explorer pane, select View menu > Solution Explorer. Select Add > New Item. In the Add New Item window, select the Report icon. Type "PatientDetail.rdl" into the Name text box. Select the Add button on the lower right side of the Add New Item dialog box to complete the process. Report Designer opens and displays the Patient Detail report file in Design view. Setup Connection In the Report Data pane, select New > Data Source. If the Report Data pane isn't visible, then select View menu > Report Data OR (ctrl + Alt + D). The Data Source Properties dialog box opens with the General section displayed. In the Name text box, type “PatientDetail". Select the Embedded connection radio button. In the Type dropdown selection box, select "Microsoft SQL Server". In the Connection string text box, type the following string: Data source=Magnusminds; initial catalog=Patient Select the Credentials tab, and under the section Change the credentials used to connect to the data source, select the Use Windows Authentication (integrated security) radio button. Select OK to complete the process. Define a Dataset for the Table Report In the Report Data pane, select New > Dataset.... The Dataset Properties dialog box opens with the Query section displayed. In the Name text box, type "GetPatientDetails". Below that, select the Use a dataset embedded in my report radio button. From the Data source dropdown box, select PatientDetail. For the Query type, select the Text radio button and Type Query into the Query text box. Select OK to exit the Dataset Properties dialog box. The Report Data pane displays the AdventureWorksDataset dataset and fields. Add a Table to the Report Select the Toolbox tab in the left pane of the Report Designer. With your mouse, select the Table object and drag it to the report design surface. Report Designer draws a table data region with three columns in the center of the design surface. If you don't see the Toolbox tab, select View menu >Toolbox. In the Report Data pane, expand the AdventureWorksDataset to display the fields. Drag the field from the Report Data pane to the first column in the table. Preview Your Report Select the Preview tab. Report Designer runs the report and displays it in the Preview view. The following diagram shows part of the report in Preview view. Deployment of An RDL Report File in SQL Report Server By Uploading RDL file in Report Server. Open SSRS Server from webportal URL. There, you will see the upload button. Click the upload option and browse the rdl file of the report from the location. It uploads your report to the report server. Click on the uploaded file it runs the report in the browser, hence, you can view it in the browser.
