Manage environment configuration in Fabric User data functions with variable libraries
Data Engineers working with Microsoft Fabric often need to manage environment-specific configurations, including the modification of Lakehouse names, file paths, or schema names for development, testing, and production environments. In this scenario, you would want to avoid hard coding this information. This is where variable libraries in Fabric can help data engineers manage their environment configuration when working with Fabric User data function.
Benefits of using variable libraries:
- Consistency: Define values once and reuse them across multiple user data functions within your workspace.
- Security: Keep sensitive data out of your code by referencing secure variables.
- Flexibility: Update configurations without redeploying code.
- Deployment pipeline integration: Variables can be injected into CI/CD workflows when working with deployment pipelines in Fabric.
Adding a variable library to a user data function
- Using Develop mode, find and select Manage connections in the ribbon of the Functions portal editor. Manage connections allows to reference to any Fabric item supported by User data functions such as Fabric SQL Database or Variable library. Learn more on what is supported.
- Select a variable library item that will be referenced in your code.
- Once created, the new connection to variable library is added. Make a note of the Alias field. You will use the alias to reference the item in your code.
NOTE: Always use the latest `fabric-user-data-functions` library.
Accessing Variable values in user data functions
To retrieve variable values stored in a library item within your function, reference the library using the alias assigned during setup. This method enhances code flexibility and security by enabling dynamic access to configuration values or secrets without the need for hardcoding.
Example
Implements a Fabric User Data Function that sends a chat request to Azure OpenAI using configuration values from Fabric Variable Library and secrets from Azure Key Vault.
Code snippet
# Select 'Manage connections' and add a connection to a Variable Library
# Replace the alias "<My Variable Library Alias>" with your connection alias.
from openai import AzureOpenAI
from azure.keyvault.secrets import SecretClient
udf = fn.UserDataFunctions()
@udf.generic_connection(argName="keyVaultClient", audienceType="KeyVault")
@udf.connection(argName="varLib", alias="<My Variable Library Alias>")
@udf.function()
def chat_request(prompt: str, keyVaultClient: fn.FabricItem, varLib: fn.FabricVariablesClient) -> str:
'''
Description: Sends a chat completion request to an Azure OpenAI model using configuration values
retrieved from a Fabric Variable Library and Azure Key Vault.
Pre-requisites:
* Create an Azure OpenAI endpoint in Azure Portal
* Create an Azure Key Vault and store your Azure OpenAI API key as a secret
* Grant your Fabric User Data Functions item owner's identity access to read secrets (Access policies or RBAC). Guidance: https://learn.microsoft.com/en-us/fabric/data-factory/azure-key-vault-reference-overview
* Create a Variable Library in Fabric and add variables for:
- KEY_VAULT_URL: Your Azure Key Vault URL (e.g., "https://your-keyvault.vault.azure.net/")
- API_KEY_SECRET_NAME: Name of the secret in Key Vault containing the API key
- ENDPOINT: Your Azure OpenAI endpoint URL
- MODEL: Your deployed model name
* Add the openai and azure-keyvault-secrets libraries to your function dependencies
* Ensure fabric-user-data-functions library is using the latest version
Args:
prompt (str): The user input or query to be processed by the model.
varLib (fn.FabricVariablesClient): A client instance to access stored variables in Variable Library
for Key Vault URL, secret name, endpoint, and model name.
Returns:
str: The generated response from the Azure OpenAI model.
'''
# Retrieve configuration from Variable Library
variables = varLib.getVariables()
key_vault_url = variables["KEY_VAULT_URL"]
api_key_secret_name = variables["API_KEY_SECRET_NAME"]
endpoint = variables["ENDPOINT"]
model_name = variables["MODEL"]
# Obtain a credential from the generic Key Vault connection (Fabric-managed identity)
credential = keyVaultClient.get_access_token()
secret_client = SecretClient(vault_url=key_vault_url, credential=credential)
key = secret_client.get_secret(api_key_secret_name).value
api_version = "2024-12-01-preview"
client = AzureOpenAI(
api_version=api_version,
azure_endpoint=endpoint,
api_key=key,
)
response = client.chat.completions.create(
messages=[
{
"role": "system",
"content": "You are a helpful assistant.",
},
{
"role": "user",
"content": prompt
}
],
max_completion_tokens=13107,
temperature=1.0,
top_p=1.0,
frequency_penalty=0.0,
presence_penalty=0.0,
model=model_name
)
return (response.choices[0].message.content)
This code does the following:
- It fetches Key Vault URL, secret name, endpoint, and model details from Variable Library. Update these values:
- KEY_VAULT_URL = https://my-keyvault.vault.azure.net/
- API_KEY_SECRET_NAME = “openai-api-key”
- ENDPOINT = https://your-resource.openai.azure.com/
- MODEL = “gpt-4”.
- Use the generic Key Vault connection to obtain an access token (managed by Fabric).
- Retrieve the API key securely from Azure Key Vault using SecretClient.
- Initialize the AzureOpenAI client with the retrieved configuration.
- Send a chat completion request with the given prompt and system instructions.
- Return the content of the first message in the response.
Conclusion
Variable libraries in User data functions ensure consistency by letting you define values once and reuse them. They improve security through referencing sensitive information securely, not directly in code. You can update configurations without redeploying and integrate variables into deployment pipelines for seamless CI/CD workflows with Fabric. Checkout these sample functions.
