Questions tagged [azure-data-factory]
Microsoft Azure Data Factory is a cloud service used to invoke (orchestrate) other Azure or external services in a controlled way using the concept of time slices. ADF also provides means to transform data in a parallel way. Data factories are predominately developed using hand crafted JSON, this provides the tool with instructions on what activities to perform.
azure-data-factory
9,428
questions
86
votes
9
answers
91k
views
The subscription is not registered to use namespace 'Microsoft.DataFactory' error
Going through this tutorial "Create a pipeline with Copy Activity using Visual Studio"
and receiving this error when I hit publish.
Creating datafactory-Name:VSTutorialFactory,Tags:,...
- 1,217
79
votes
11
answers
160k
views
The client with object id does not have authorization to perform action 'Microsoft.DataFactory/datafactories/datapipelines/read' over scope
I was trying to invoke data factory pipeline from azure function programmatically. Its throwing following error.
link:
http://eatcodelive.com/2016/02/24/starting-an-azure-data-factory-pipeline-...
- 931
28
votes
2
answers
30k
views
How to escape a single quote in a dynamic expression with functions?
Need to insert a parameter into a string using the dynamic data function from the pipeline parameters. Have tried backslash, double backslash, double single quote,@, @@, and other such nonsense. ...
- 973
26
votes
2
answers
9k
views
Azure Data Factory project with Visual Studio 2017
I am not sure whether Azure Data Factory project is supported on Visual Studio 2017 at the moment.
I have just installed VS 2017 but cannot open our solution as there is one azure data factory ...
- 74.9k
21
votes
4
answers
39k
views
How to run SQL Script in Azure Data Factory v2?
There is NO Sql Script activity in Azure Data Factory V2. So how can I create a stored proc, a schema in a database? What are my options?
- 4,413
21
votes
1
answer
22k
views
What is the difference between ADF Pipeline and ADF Data flow?
What is the difference between ADF Pipeline and ADF Data flow? Why are sinks/sources supported in Pipeline and Data flow different? Is it possible to create a pipeline that reads data from source, ...
- 3,470
19
votes
3
answers
23k
views
Variables and Parameters in Azure Data Factory?
Im just learning ADF, can you explain to me the difference between Variables and Parameters? I cant find a good explanation about it.
- 227
19
votes
7
answers
19k
views
Disable activity in Azure Data factory pipeline without removing it
So I am testing each of the activities of the pipeline and I want to disable some of the activities in it. Essentially there is an activity of sending emails which I want to disable as I wanted to see ...
- 1,885
16
votes
2
answers
15k
views
Azure Data Factory v2: Activity execute pipeline output
Is there a way to reference the output of an executed pipeline in the activity "Execute pipeline"?
I.e.: master pipeline executes 2 pipelines in sequence. The first pipeline generates an own ...
- 173
15
votes
2
answers
16k
views
Difference between DataFlow and Pipelines
I do not understand the difference between dataflow and pipeline in Azure Data Factory.
I have read and see DataFlow can Transform Data without writing any line of code.
But I have made a pipeline ...
- 8,823
14
votes
2
answers
10k
views
How to get custom output from an executed pipeline?
I would like to be able to get custom output from an "Execute Pipeline Activity". During the execution of the invoked pipeline, I capture some information in a variable using the "Set Variable" ...
- 362
13
votes
3
answers
13k
views
Azure Data Factory specify custom output filename when copying to Blob Storage
I'm currently using ADF to copy files from an SFTP server to Blob Storage on a scheduled basis.
The filename structure is AAAAAA_BBBBBB_CCCCCC.txt.
Is it possible to rename the file before copying ...
- 1,378
13
votes
3
answers
6k
views
How to transform xml data using datafactory pipeline
How do we save data inside of an XML payload to blob storage?
input
<root>
<alexIsAwesome>yes he is</alexIsAwesome>
<bytes>sdfsdfjijOIJOISJDFQPWORPJkjsdlfkjlksdf==</...
- 59.3k
12
votes
2
answers
574
views
Moving a DocumentDB Collection to Azure Data Lake Storage
I was wondering what's the best practice moving a documentDB to the Azure Data Lake Storage.
Should I create a file for each document in a collection or move the entire documentDB?
Also I didn't find ...
- 3,797
11
votes
8
answers
19k
views
Azure Data Factory V2: Optional parameters
I'm using ADFv2 and I would like to use some optional parameters defined at the dataset level. I didn't found if that can be achieved.
If I define a parameter in the dataset (after defining a ...
- 446
11
votes
6
answers
15k
views
How to get the Azure Data Factory parameters into the ARM template parameters file (ARMTemplateParametersForFactory.json) after publishing
I am trying to create my Azure DevOps release pipeline for Azure Data Factory.
I have followed the rather cryptic guide from Microsoft (https://learn.microsoft.com/en-us/azure/data-factory/continuous-...
- 1,909
11
votes
3
answers
14k
views
how to change Data Factory in Microsoft Integration Runtime COnfiguration Manager?
I have Installed Microsoft Integration Runtime configuration Manager When I have Migrated Data from On-Premise SQL Server to Azure Data Lake and when I'm trying to use for another Azure Data Factory I ...
- 430
11
votes
1
answer
15k
views
Azure Data Factory CSV with double quotes
I have a pipeline that retrieves an FTP hosted CSV file. It is comma delimited with double quote identifiers. The issue exists where a string is encapsulated in double quotes, but the string itself ...
- 117
11
votes
4
answers
26k
views
Copy Data From Azure Blob Storage to AWS S3
I am new to Azure Data Factory and have an interesting requirement.
I need to move files from Azure Blob storage to Amazon S3, ideally using Azure Data Factory.
However S3 isnt supported as a sink;...
- 1,273
11
votes
4
answers
20k
views
Import python module to python script in databricks
I am working on a project in Azure DataFactory, and I have a pipeline that runs a Databricks python script. This particular script, which is located in the Databricks file system and is run by the ADF ...
- 691
11
votes
5
answers
14k
views
Azure Data Factory and SharePoint
I have some Excel files stored in SharePoint online. I want copy files stored in SharePoint folders to Azure Blob storage.
To achieve this, I am creating a new pipeline in Azure Data factory using ...
- 111
11
votes
2
answers
32k
views
How to decompress a zip file in Azure Data Factory v2
I'm trying to decompress a zip file (with multiple files inside) using Azure Data Factory v2. The zip file is located in Azure File Storage. The ADF Copy task just copies the original zip file without ...
- 111
10
votes
6
answers
9k
views
How do I raise an error from if activity in ADFV2?
I have a child pipeline that receives instructions from a control file using a dataset. These instructions define what directory to copy files from.
First this child pipeline checks that a file ...
- 1,641
10
votes
1
answer
26k
views
Azure Data Factory check rowcount of copied records
I am designing a ADF pipeline that copies rows from a SQL table to a folder in Azure Data Lake. After that the rows in SQL should be deleted. But for this delete action takes place I want to know if ...
- 303
9
votes
1
answer
14k
views
Data Factory, Synapse Analytics and DataBricks Comparison
I'm little bit new on Azure and I'm wondering when is recommendable to use ADF, Synapse, or DataBricks. What are their use cases for best practices and performance?
Could you help me with this ...
- 413
9
votes
2
answers
13k
views
How can I get the last day of a month in dynamic content in ADF2?
I want to get the last day of a month based on the utcnow() timestamp.
Instead of "dd" in the expression bellow there should be automatically the last day of the month (28, 30 or 31):
@{...
- 491
9
votes
1
answer
19k
views
How to create and set a variable in azure data factory v2
How do I set an variable of type array of strings in ADF v2 so that I can iterate over each element? Each element represents a database name?
The 'Set Variable' activity has a dropdown but displays '...
- 747
9
votes
1
answer
13k
views
How to increment a parameter in an Azure Data Factory Until Activity?
I am accessing a RESTful API that pages results in groups of 50 using the HTTP connector. The REST connector doesn't seem to support Client Certificates so I can't use the pagination in that.
I have ...
- 7,686
9
votes
4
answers
15k
views
Azure: Compare dates in Data factory
Data factory doesn't have a built-in date difference function. I want to compare lastModified date and utcnow in if condition activity. How can I achieve it?
@greaterOrEquals(activity('Get Metadata1')...
- 1,356
9
votes
2
answers
25k
views
How to use the pre-copy script from the copy activity to remove records in the sink based on the change tracking table from the source?
I am trying to use change tracking to copy data incrementally from a SQL Server to an Azure SQL Database. I followed the tutorial on Microsoft Azure documentation but I ran into some problems when ...
9
votes
2
answers
9k
views
Can an System assigned managed service identity be added to an AAD group?
I have an Azure Data Factory V2 service running with an MSI identity. This service needs to access a Data Lake Gen 1 with thousands of folders and millions of files.
For efficiency, we have a group ...
- 1,641
9
votes
1
answer
10k
views
Failed to check the resource group status: 403 while deploying ARM template using CD pipeline through service principal/connection
While deploying Azure Data Factory's ARM template through service principal, I am getting below error:
[error]Failed to check the resource group status. Error: {"statusCode":403}.
Is it a ...
9
votes
2
answers
16k
views
Azure Data Factory connecting to Blob Storage via Access Key
I'm trying to build a very basic data flow in Azure Data Factory pulling a JSON file from blob storage, performing a transformation on some columns, and storing in a SQL database. I originally ...
- 838
9
votes
2
answers
2k
views
Secure ADF v2 Pipeline Parameter String for WebActivity
I have ADF v2 Pipeline with a WebActivity which has a REST Post Call to get Jwt Access token from AD token api (https://login.microsoftonline.com/myorg.onmicrosoft.com/oauth2/token)
I have to pass ...
9
votes
0
answers
4k
views
Azure Data Factory 2, Data flow with derived column, how to set a timestamp to null
I have a Azure DataFactory V2 pipeline that executes a data flow.
The Azure Sql source table have a Date not null column, DateReported.
This is imported from an external source. Where null values are
...
- 356
8
votes
1
answer
21k
views
pass parameters to SQL query in Azure data factory
How can we pass parameter to SQL query in Azure data factory ...
ex: Select * from xyz_tbl where date between @date1 and @date2
if I use stored procedure with output @date1 and @date2 parameter, how ...
- 510
8
votes
5
answers
15k
views
Azure Data Factory - How can I trigger Scheduled/OneTime Pipelines?
Background : I have scheduled pipelines running for copying data from source to destination. This is scheduled to run daily at a specific time.
Problem : The input dataset to the pipeline is ...
- 167
8
votes
5
answers
2k
views
Azure data factories vs factory
I'm building an Azure data lake using data factory at the moment, and am after some advice on having multiple data factories vs just one.
I have one data factory at the moment, that is sourcing data ...
- 203
8
votes
6
answers
29k
views
Using Azure Data Factory to get data from a REST API
Is it possible to use Azure Data Factory to get data from a REST API and insert it to a Azure database table?
- 651
8
votes
2
answers
11k
views
How to do undo in Azure Data Factory
I am new to Azure Data factory. While developing the pipeline I could not find undo operation in Azure Data Factory. ctrl+z did not work. What is the keyboard shortcut for the undo?
- 81
8
votes
2
answers
18k
views
how to export pipeline in datafactory v2 or migrate to another
I'm trying export one pipeline created in datafactory v2 or migrate to another, but not found the option,
Could you help me please
8
votes
5
answers
5k
views
ARM Template - auto approval of managed private endpoint
I am developing an ARM template for Azure Data Factory with managed private endpoints to SQL Server and Azure Datalake. However, when the ARM template completes execution, the managed private ...
- 1,045
8
votes
4
answers
10k
views
Azure Data Factories vs SSIS
I am thinking of moving our SSIS ETLs to Azure Data Factory. My arguments in favour of such leap are:
Our sources and targets are already in the cloud. ADF is cloud native so it seems at good fit.
...
- 81
8
votes
2
answers
37k
views
How to use formatDateTime in Azure Data Factory?
I would like to format the pipeline trigger time in this format:
10-Mar-2021 08:31:59 AM
Here is the code I am using:
@formatDateTime(pipeline().TriggerTime, 'DD-MON-YYYY HH:MI:SS AM/PM')
However ...
- 769
8
votes
5
answers
12k
views
Azure Data Factory pipelines are failing when no files available in the source
Currently – we do our data loads from Hadoop on-premise server to SQL DW [ via ADF Staged Copy and DMG on-premise server]. We noticed that ADF pipelines are failing – when there are no files in the ...
- 189
7
votes
2
answers
15k
views
Can we pass Databricks output to function in an ADF Job?
Can anyone help me with Databricks and Azure function?
I'm trying to pass data bricks JSON output to azure function body in ADF job, is it possible?
If yes, How?
If No, what other alternatives to do ...
- 151
7
votes
3
answers
5k
views
Azure Data Factory - Error creating a parameterised mapping dataflow
I am having an error when trying to create a parameterised Mapping Data Flow. More specifically, I have the following error. Anyone have suggestions on how to fix it or what the error may be?
{ "...
- 241
7
votes
3
answers
19k
views
azure data factory: use variables in query
I have created a copy activity that copies data from an on premise database to a Azure SQL Database.
I need to modify dynamiccaly the query, so it takes a range of dates, with these two variables:
...
- 1,917
7
votes
1
answer
14k
views
MERGE data in a Dataflow of Azure Data Factory into an existing table
I am trying to create a DataFlow under Azure Data Factory that inserts & updates rows into a table after performing some transformations. When I am trying to write the modified data into a 'Sink' ...
- 2,631
7
votes
1
answer
5k
views
Difference between deployment and provisioning ADF
Whenever I create a new pipeline in ADF, and click on 'deploy', at first I see the message saying "deplyoing", right after that I see the message saying "provisioning". What's the difference?
Can we ...
- 3,582