Egil Hansen's .com

Blazor Server: Stream large amount of data to JavaScript with cancellation

Wed, 04 Sep 2024 00:00:00 +0000

In a recent Blazor Server (.NET 8) project I needed to send a large amount of data to Plotly.js (+ 50 MB of data). Blazor already have support for streaming data from .NET to JavaScript, however, I wanted to also add support for compression of the data that is sent and cancellation of an ongoing stream of data, in case the user changes their mind and wants to view a different set of data, before the current set of data has been sent.

To eanble compression and cancellation, the following is needed:

Custom JavaScript that can handle decompression and cancellation on the client side of things (streamingDataJsInterop.js).
A new .NET type that handles the serialization and compression of data, and cancellation from the server (StreamingDataJsInterop.cs).

The StreamingDataJsInterop type will allow one active stream of data to be send at the time. Sending new data cancelled any active data streaming before sending the new data. If you want multiple streams of data concurrently, just instantiate as many StreamingDataJsInterop as you need.

Setup

Include the streamingDataJsInterop.js in your wwwroot and add a script reference in your App.razor after _framework/blazor.web.js, e.g.:

<script src="_framework/blazor.web.js"></script>
<script src="streamingDataJsInterop.js"></script>

Then, add the StreamingDataJsInterop.cs to your project, and now you can use it in your components, e.g.:

@implements IDisposable
@inject IJSRuntime JSRuntime

<!-- ... --> 

@code {
    private StreamingDataJsInterop streamingDataJsInterop = default!;

    protected override void OnInitialized()
    {
        streamingDataJsInterop = new StreamingDataJsInterop(JSRuntime);
    }

    public void Dispose()
    {
        streamingDataJsInterop.Dispose();
    }

    private async Task SendStreamingData()
    {
        var data = // get data from somewhere 
        try
        {
            await streamingDataJsInterop.InvokeStreamingAsync(
                "receiveStreamingData",
                data);
        catch (OperationCanceledException)
        {
            // sending was cancelled...
        }
    }
}

Then, add the JavaScript function that should be receive the streamed data, e.g.:

async function receiveStreamingData(streamRef) {
    try {
        const data = await streamRef.getData();
    }
    catch (err) {
        // streaming was cancelled from the server.
    }
}

There is a complete sample available on here: github.com/egil/BlazorServerStreamingToJavaScript.

Hope this help!

SSIS: Using ASCII control characters anywhere in a SSIS package will break it

Thu, 13 Aug 2015 00:00:00 +0000

I noticed this oddity when I was reading a CSV file using SSIS. Sometimes the source file would have a the ASCII SUB control character in the first column on the last line of the file (see screenshot below):

My first attempt to remove the line from Data Flow pipeline was to add a Conditional Split transformation to the Data Flow task. There, I checked if the first column was equal to the character, as seen in the screenshot below.

The editor did not complain, however, when I tried to execute the task, nothing happened. After few frustrating minutes I tried to restart Visual Studio, but when I reopened my package and looked in my Data Flow task, it was completely empty.

My guess is that adding any ASCII control character, i.e. non-printing character, to somewhere in a SSIS package will caused the XML engine in Visual Studio to choke, and the result was that nothing was saved to my packaged .dtsx file.

So how do we use ASCII control characters in SSIS. The right (and now obvious) approach is to use the CODEPOINT and lookup number of the ASCII control character we want to match with in an ASCII table, e.g.:

Hope this help!

SSIS: Lookup Transform not finding expected match with NULL values

Wed, 12 Aug 2015 00:00:00 +0000

If you use the Lookup Transform to perform lookups with columns that can be NULL, the Lookup Transform will fail to find expected matches because in a database world, NULL does not equal NULL.

So how do we make the Lookup Transform match as expected when source column and lookup column can be NULL? We use ISNULL function, partial caching and a custom query in the Lookup Transform.

Let us look at a quick example. This is our lookup table definition, which allows NULLs in the LastName column, that we want to get the Id of a person from, based on their first and last name:

CREATE TABLE Person (
  Id int IDENTITY(1,1) NOT NULL,
  FirstName varchar(50) NOT NULL,
  LastName varchar(50) NULL
)

In SSIS, we have a input source with a FirstName column and a LastName, where the LastName column might sometimes be NULL. To find the Id of each name, we add a Lookup Transform to the Data Flow Task, and configure it to use Partial cache in the General tab. In the Connection tab we point to our Person table as normal, and in the Columns tab we map our input’s FirstName column and LastName column to the lookup’s FirstName and LastName columns.

Until now, every step has been the same as with Full caching. However, now we continue to the Advanced tab and check the *Modify the SQL statement checkbox and change the SQL like so:

select * from (select * from [Person]) [refTable]
where [refTable].[FirstName] = ? and ISNULL([refTable].[LastName], '') = ISNULL(?, '')

With the help of the ISNULL function, we rewrite any NULL values to an empty string, which is comparable, but only if the LastName is actually NULL.

Last step is to click the Parameters button and assign the parameters.

Hope this help!

SSIS: Solution for Excel data source text truncation of column with more than 255 characters

Tue, 28 Jul 2015 00:00:00 +0000

When using SQL Server Integration Service (SSIS) Excel data source to import text from a column in a spreadsheet, the data source will automatically try to determine the maximum length of text in the text column. It does so by sampling the 9 first rows in the spreadsheet and measuring their length. If it does not find any column longer than 255 characters, it will set the columns type as DT_WSTR with a length of 255, otherwise it will use DT_NTEXT (see Excel Source on TechNet for more details).

The obvious problem is that if any row after 9th row is longer than 255 characters, the import will break with the error message: “Text was truncated or one or more characters had no match in the target code page.”

The suggested solution on TechNet is to change a registry key to increase the number of rows the Excel data source will sample to determine the length of a text column. Other suggestions online include changing the spreadsheet to have a larger row in the top 8 rows. However, neither solution is maintainable if you cannot change the source spreadsheet or if it is a new version of the spreadsheet each time you execute your SSIS package.

My solution is based on using the Script Component instead of the Excel data source. It allows you to define each column’s type and size/length as you do with the Flat File Connection Manager. The only other requirement is that there is a header row in the sheet you are trying to import from.

The solution descibed below has been tested with SSIS 2012 but should work with future versions.

Solution

To help guide us through the solution, we will use this very simple Excel spreadsheet as our example:

Add an Excel Connection Manager to the package as you normally would. Point it to your source file, choose an Excel version, and check the “First row has column names” checkbox.

Note: This particular solution assums the first row in the Excel spreadsheet is a header row. This solution can be modified to work without, but not without changes to the script code.
Add a Script Component as a Source and open the Script Component editor (Script Transformation Editor)
Add your Excel Connection Manager in the “Connection Managers” tab. Make sure to name it DataSourceConnection, we will use that name in the script code later.
In the “Inputs and Outputs” tab, first rename the default “Output 0” to the exact name of the sheet in your Excel spreadsheet you want to import data from.
Then continue adding the columns you want to export to the output, also naming each column with the exact name it has in the spreadsheet and pick the desired type.
Note: The names of the columns and sheet has to be the same since we use them to automatically generate the OleDB select statement that will read the data from the spread sheet.
Go to the Script tab and click the “Edit Script…” button. This will open a new instance of Visual Studio.
In the Visual Studio script edit, make the following changes to the script file:

Add the following using statements to the top of the file:

using System.Data.OleDb;
using System.Text;

At the top of the ScriptMain class, add these three lines:

private OleDbConnection connection;
private OleDbCommand command;
private OleDbDataReader reader;

Replace the PreExecute() and PostExecute() methods with the following new versions:

public override void PreExecute()
{
  base.PreExecute();

  connection = new OleDbConnection(Connections.DataSourceConnection.ConnectionString);
  connection.Open();
  command = new OleDbCommand();
  command.Connection = connection;

  // Build select query by iterating over the columns in first output collection
  var query = new StringBuilder("SELECT ");
  var outputCollection = ComponentMetaData.OutputCollection[0];

  for (int i = 0; i < outputCollection.OutputColumnCollection.Count; i++)
  {
    query.AppendFormat(" [{0}]", outputCollection.OutputColumnCollection[i].Name);
    if (i < outputCollection.OutputColumnCollection.Count - 1) query.Append(",");
  }

  query.AppendFormat(" FROM [{0}]", outputCollection.Name);

  command.CommandText = query.ToString();
  reader = command.ExecuteReader();
}

public override void PostExecute()
{
  base.PostExecute();

  command = null;
  reader.Close();
  connection.Close();
}

Finally you need to do a little coding yourself in the CreateNewOutputRows method. Here, you parse each column in each row and add it to the output buffer. The following is the code used in the example:

public override void CreateNewOutputRows()
{
  while (reader.Read())
  {
    // Replace with own import code matching the name
    // of the import buffer defined in the Script Transformation Editor
    // and column names
    NumbersandTextBuffer.AddRow();

    NumbersandTextBuffer.TimesUsed = (double)reader["Times Used"]
    NumbersandTextBuffer.TextShown = (string)reader["Text Shown"];
  }                     
}

Once you are done tweaking the CreateNewOutputRows method, close the Visual Studio script editor instance and you are done. The complete ScriptMain example file is posted below for reference.

Hope this helps!

using System;
using System.Data;
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Runtime.Wrapper;
using System.Data.OleDb;
using System.Text;

[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{
  private OleDbConnection connection;
  private OleDbCommand command;
  private OleDbDataReader reader;

  public override void PreExecute()
  {
    base.PreExecute();
 
    connection = new OleDbConnection(Connections.DataSourceConnection.ConnectionString);
    connection.Open();
    command = new OleDbCommand();
    command.Connection = connection;

    // Build select query by iterating over the columns in first output collection
    var query = new StringBuilder("SELECT ");
    var outputCollection = ComponentMetaData.OutputCollection[0];

    for (int i = 0; i < outputCollection.OutputColumnCollection.Count; i++)
    {
      query.AppendFormat(" [{0}]", outputCollection.OutputColumnCollection[i].Name);
      if (i < outputCollection.OutputColumnCollection.Count - 1) query.Append(",");
    }

    query.AppendFormat(" FROM [{0}]", outputCollection.Name);

    command.CommandText = query.ToString();
    reader = command.ExecuteReader();
  }

  public override void PostExecute()
  {
    base.PostExecute();

    command = null;
    reader.Close();
    connection.Close();
  }

  public override void CreateNewOutputRows()
  {
    while (reader.Read())
    {
      // Replace with own import code matching the name
      // of the import buffer defined in the Script Transformation Editor
      // and column names
      NumbersandTextBuffer.AddRow();

      NumbersandTextBuffer.TimesUsed = (double)reader["Times Used"]
      NumbersandTextBuffer.TextShown = (string)reader["Text Shown"];
    }                     
  }
}

SharePoint 2013: The server was unable to save the form at this time

Wed, 14 Aug 2013 00:00:00 +0000

This morning our freshly installed SharePoint Server 2013 server was refusing to let me save a simple form with a Lookup column to another list. Using the standard insert/edit form, the error message read:

Error message from insert/edit form: The server was unable to save the form at this time. Please try again.

When using the quick edit mode, the error message was:

Error message in quick edit mode: This service isn’t available right now. Click here to try again.

The fix turned out to be a iisreset on the SharePoint server.

Update: It looks like the culprit is the Security Token Service, more info over at Sharepoint2013.me.

I hope this helps somebody else save a few hours of head scratching.

Extra Tips for Debugging SharePoint Timer Jobs

Mon, 15 Jul 2013 00:00:00 +0000

There is an excellent guide over on MSDN on how to debug a SharePoint timer job from Visual Studio. However, it fails to mention that sometimes you need to restart the service that runs the timer jobs to allow debugger to set a breakpoint. In addition, sometimes a job will not show up in the job definitions list (Central Administration – Job Definitions), making it hard to manually start a job. The solution to this is to recycle IIS.

The simplest way to incorporate this into a development workflow is to add it to these commands to the Pre- and Post-deployment Command Line steps in Visual Studio:

Pre-deployment Command Line:

net stop SPTimerV4

Post-deployment Command Line:

net start SPTimerV4
iisreset

Hope this helps.

SharePoint Foundation 2010 on Windows 7/8 – missing config.xml file

Tue, 09 Jul 2013 00:00:00 +0000

To install SharePoint Foundation 2010 on Windows 7/8, you need to modify the config.xml file. However, after extracting the setup files, the config.xml file was nowhere to be seen in the Setup folder. To get the installation going, I had to create the file myself. Here is the content that goes into it:

<Configuration>
   <Package Id="sts">
      <Setting Id="SETUPTYPE" Value="CLEAN_INSTALL"/>
   </Package>
   <DATADIR Value="%CommonProgramFiles%\Microsoft Shared\Web Server Extensions\14\Data"/>
   <Logging Type="verbose" Path="%temp%" Template="Microsoft SharePoint Foundation 2010 Setup *.log"/>
   <Setting Id="UsingUIInstallMode" Value="1"/>
   <Setting Id="SETUP_REBOOT" Value="Never"/>
   <Setting Id="AllowWindowsClientInstall" Value="True"/>
</Configuration>

And to force the setup.exe to use it, we need to apply the /config switch when starting setup.exe, i.e.: setup.exe /config Setup\config.xml

Other useful links for installing SharePoint Foundation 2010 on Windows 8:

SharePoint 2010 Management Shell does not load with Windows PowerShell 3.0 - because SharePoint PowerShell does not work out of the box.
Develoq.net: Installing Sharepoint Foundation 2010 on Windows 7 - how to install on Farm mode on Windows 7/8, to avoid getting another SQL server instance.
Installing SharePoint 2010 on Windows 8 - Hints about getting installation to work on Windows 8, also good tips in comments.

Hope this helps.

JSLink: The simple way to add and load multiple JavaScript files

Mon, 11 Mar 2013 00:00:00 +0000

It can sometimes be useful to specify multiple JavaScript files to use when rendering e.g. a field in SharePoint 2013. It could be that you have a JavaScript file with some shared code that you use in all your custom rendering, or you use a library such as jQuery. In either case, you need all the scripts loaded before you can start rendering.

It turns out it is easy to tell SharePoint to download multiple JavaScript files when rendering e.g. a field. Just separate them with the pipe character, i.e. |.

For example, to load ~sitecollection/_catalogs/masterpage/jQuery.js and ~sitecollection/_catalogs/masterpage/my-custom-field.js, set the JSLink property of your target to:

JSLink="~sitecollection/_catalogs/masterpage/jQuery.js|~sitecollection/_catalogs/masterpage/my-custom-field.js"

Do not worry about SharePoint re-downloading the same script repeatedly if you use it to render multiple elements, SharePoint will not download a script more than one time (e.g. jQuery).

How to add an Enterprise Keywords column to a custom list and enable Keyword synchronization in SharePoint 2013

Sat, 09 Mar 2013 00:00:00 +0000

This is a quick step-by-step guide to add an Enterprise Keywords column to a custom list and enable Keyword synchronization with Visual Studio 2012 in a SharePoint project. The SharePoint tools for Visual Studio 2012 have become a lot better in the latest edition, but a little extra work is required to get the Enterprise keywords column wired up correctly.

Step 1: Create an empty SharePoint 2013 project.

Step 2: Add Content Type and add these three columns to it manually through the Elements.xml file or through the graphical interface:

<FieldRef ID="{23f27201-bee3-471e-b2e7-b64fd8b7ca38}" 
	DisplayName="$Resources:osrvcore,field_KeywordsFieldName" 
	Required="FALSE" 
	Description="$Resources:osrvcore,field_KeywordsFieldDesc" 
	Hidden="FALSE" 
	Name="TaxKeyword" 
	Sealed="TRUE" 
	Sortable="FALSE" />
<FieldRef ID="{1390a86a-23da-45f0-8efe-ef36edadfb39}"
	DisplayName="TaxKeywordTaxHTField" 
	Required="FALSE" 
	Hidden="TRUE" 
	Name="TaxKeywordTaxHTField" />
<FieldRef ID="{f3b0adf9-c1a2-4b02-920d-943fba4b3611}" 
	DisplayName="Taxonomy Catch All Column" 
	Required="FALSE" 
	Hidden="TRUE" 
	Name="TaxCatchAll" 
	Sealed="TRUE" 
	Sortable="FALSE" />

Step 3: Add a List to the project; add the custom content type you created previously content. Alternatively, if you already have an existing list, add these fields definitions inside the <Fields> tag in lists Schema.xml file:

<Field Type='TaxonomyFieldTypeMulti'
  DisplayName='Enterprise Keywords'
  StaticName='TaxKeyword'
  Name='TaxKeyword'
  ID='{23f27201-bee3-471e-b2e7-b64fd8b7ca38}'
  List="Lists/TaxonomyHiddenList"
  WebId='~sitecollection'
  ShowInViewForms='TRUE'
  DefaultListField='TRUE'
  Required='FALSE'
  Hidden='FALSE'
  ShowField="Term$Resources:core,Language;"
  Mult='TRUE'
  Sortable='FALSE'
  Group="$Resources:osrvcore,field_KeywordsGroupName"
  Description="$Resources:osrvcore,field_KeywordsFieldDesc"
  AllowDeletion='TRUE'>
  <Customization>
    <ArrayOfProperty>
      <Property>
        <Name>SspId</Name>
        <Value
          xmlns:q1='http://www.w3.org/2001/XMLSchema' p4:type='q1:string'
          xmlns:p4='http://www.w3.org/2001/XMLSchema-instance'>00000000-0000-0000-0000-000000000000</Value>
      </Property>
      <Property>
        <Name>GroupId</Name>
      </Property>
      <Property>
        <Name>TermSetId</Name>
        <Value
          xmlns:q2='http://www.w3.org/2001/XMLSchema' p4:type='q2:string'
          xmlns:p4='http://www.w3.org/2001/XMLSchema-instance'>00000000-0000-0000-0000-000000000000</Value>
      </Property>
      <Property>
        <Name>AnchorId</Name>
        <Value
          xmlns:q3='http://www.w3.org/2001/XMLSchema' p4:type='q3:string'
          xmlns:p4='http://www.w3.org/2001/XMLSchema-instance'>00000000-0000-0000-0000-000000000000</Value>
      </Property>
      <Property>
        <Name>UserCreated</Name>
        <Value
          xmlns:q4='http://www.w3.org/2001/XMLSchema' p4:type='q4:boolean'
          xmlns:p4='http://www.w3.org/2001/XMLSchema-instance'>false</Value>
      </Property>
      <Property>
        <Name>Open</Name>
        <Value
          xmlns:q5='http://www.w3.org/2001/XMLSchema' p4:type='q5:boolean'
          xmlns:p4='http://www.w3.org/2001/XMLSchema-instance'>true</Value>
      </Property>
      <Property>
        <Name>TextField</Name>
        <Value
          xmlns:q6='http://www.w3.org/2001/XMLSchema' p4:type='q6:string'
          xmlns:p4='http://www.w3.org/2001/XMLSchema-instance'>{1390a86a-23da-45f0-8efe-ef36edadfb39}</Value>
      </Property>
      <Property>
        <Name>IsPathRendered</Name>
        <Value
          xmlns:q7='http://www.w3.org/2001/XMLSchema' p4:type='q7:boolean'
          xmlns:p4='http://www.w3.org/2001/XMLSchema-instance'>false</Value>
      </Property>
      <Property>
        <Name>IsKeyword</Name>
        <Value
          xmlns:q8='http://www.w3.org/2001/XMLSchema' p4:type='q8:boolean'
          xmlns:p4='http://www.w3.org/2001/XMLSchema-instance'>true</Value>
      </Property>
      <Property>
        <Name>TargetTemplate</Name>
      </Property>
      <Property>
        <Name>CreateValuesInEditForm</Name>
        <Value
          xmlns:q9='http://www.w3.org/2001/XMLSchema' p4:type='q9:boolean'
          xmlns:p4='http://www.w3.org/2001/XMLSchema-instance'>false</Value>
      </Property>
      <Property>
        <Name>FilterAssemblyStrongName</Name>
      </Property>
      <Property>
        <Name>FilterClassName</Name>
      </Property>
      <Property>
        <Name>FilterMethodName</Name>
      </Property>
      <Property>
        <Name>FilterJavascriptProperty</Name>
      </Property>
    </ArrayOfProperty>
  </Customization>
</Field>
<Field Type="Note"
  DisplayName="TaxKeywordTaxHTField"
  StaticName="TaxKeywordTaxHTField"
  Name="TaxKeywordTaxHTField"
  ID="{1390a86a-23da-45f0-8efe-ef36edadfb39}"
  ShowInViewForms="FALSE"
  Required="FALSE"
  Hidden="TRUE"
  CanToggleHidden="TRUE" />

<Field Type="LookupMulti"
  DisplayName="Taxonomy Catch All Column"
  StaticName="TaxCatchAll"
  Name="TaxCatchAll"
  ID="{f3b0adf9-c1a2-4b02-920d-943fba4b3611}"
  ShowInViewForms="FALSE"
  Required="FALSE"
  Hidden="TRUE"
  ShowField="CatchAllData"
  Mult="TRUE"
  List="Lists/TaxonomyHiddenList"
  WebId="~sitecollection"
  Sortable="FALSE"
  AllowDeletion="TRUE" 
  Sealed="TRUE" />

At this point, we are usually done. The columns were added to the list definition and the columns does indeed show up in the list. However, they keywords added to items in the list does not show up. What is missing is the event receiver that takes care of the keyword synchronization. We will add that now.

Step 4: Add an Event Receiver to the project. Make sure to select the list you just created as the event source when going through the wizard, and any events in the events list (we will replace the code generated, so it does not matter which).

Aside: I like to add event receivers beneath the list they are associated with so it is clear that what list the event receiver belongs to, but anywhere inside the project is fine.

Step 5: Delete the event receiver’s code file (.cs or .vb) so you are only left with the Elements.xml file.

Step 6: Open the event receiver’s Elements.xml file and replace whatever <Receiver> tags where generated with these:

<Receiver>
	<Name>TaxonomyItemSynchronousAddedEventReceiver</Name>
	<Synchronization>Synchronous</Synchronization>
	<Type>ItemAdding</Type>
	<SequenceNumber>10000</SequenceNumber>
	<Assembly>Microsoft.SharePoint.Taxonomy, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c</Assembly>
	<Class>Microsoft.SharePoint.Taxonomy.TaxonomyItemEventReceiver</Class>
	<Data></Data>
</Receiver>
<Receiver>
	<Name>TaxonomyItemUpdatingEventReceiver</Name>
	<Synchronization>Synchronous</Synchronization>
	<Type>ItemUpdating</Type>
	<SequenceNumber>10000</SequenceNumber>
	<Assembly>Microsoft.SharePoint.Taxonomy, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c</Assembly>
	<Class>Microsoft.SharePoint.Taxonomy.TaxonomyItemEventReceiver</Class>
	<Data></Data>
</Receiver>

The important thing to make sure is that the value of the ListTemplateId attribute on the <Receivers ListTemplateId="xxxxx"> ... </Receivers> tag is the same as the value of the Type="xxxxx" attribute on the < ListTemplate ... /> tag. Otherwise the event receivers will not attach to the right list.

That is it. If you need a guide for SharePoint 2010, please take a look at this blog post over at metaengine.com. It helped me get this working in SharePoint 2013.

Hope this helps.

Nokia, please turn my f…ing volume back up again!

Sun, 03 Mar 2013 00:00:00 +0000

With the Windows Phone 7.8 update to my Nokia Lumia 800, Nokia suddenly decided that my hearing is precious and I am not able to take care of it myself… the volume is now capped at half of what it was before.

I have been using the earbuds that came with the phone. They are quite good, they do not block out all of the sound from the surroundings, and since I mostly listen to podcasts or audiobooks when travelling around the city, I always had the volume at 30 (the maximum) to be able to hear what is being said. Now I have to use earbuds that are much more uncomfortable to hear anything over the city noise.

It might be too much to ask to get both the new tile sizes and the volume I am used to, but I am going to take the chance anyway and say Nokia, please turn my f…ing volume back up again. It is after all just a registry setting…