FluentETL - Data automation made easy for coders

Created Unassigned: Consider scenarios and API for using SqlBulkCopy instead of SqlDataReader for MS SQL Server databases [1965]

grand — Mon, 20 May 2013 19:27:55 GMT

Consider scenarios and API for using SqlBulkCopy instead of SqlDataReader for MS SQL Server databases.
-SqlBulkCopy works orders of magnitude faster than SqlDataReader

Created Unassigned: Allow LINQ queries to execute on server side by adding ObjectSource.From(IQueryable) overload [1964]

grand — Mon, 20 May 2013 19:10:22 GMT

Allow LINQ queries to execute on server side by adding ObjectSource.From(IQueryable) overload.

Any code that accesses dataSource parameter of ObjectSource.From(IQueryable) should not cast it to IEnumerable or use ToList(), ToArray() or AsEnumerable() on it. This way LINQ will run server side.

Going one step further it may be better not to enumerate IQueryable dataSource parameter at all, but rather extract SQL from the parameter and pass it into SqlSource.FromQuery().

Source code checked in, #19162

Project Collection Service Accounts — Mon, 01 Oct 2012 20:50:51 GMT

Upgrade: New Version of LabDefaultTemplate.xaml. To upgrade your build definitions, please visit the following link: http://go.microsoft.com/fwlink/?LinkId=254563

Source code checked in, #19161

Project Collection Service Accounts — Mon, 01 Oct 2012 20:49:54 GMT

Checked in by server upgrade

Reviewed: FluentETL 0.2 (Dec 16, 2011)

BentoDS — Fri, 16 Dec 2011 10:58:56 GMT

Rated 5 Stars (out of 5) - Very nice and clean

Reviewed: FluentETL 0.2 (Feb 08, 2011)

Ewok — Tue, 08 Feb 2011 14:26:22 GMT

Rated 5 Stars (out of 5) - Great work. Excellant when you want to quickly get data from somewhere without having to mess about with SSIS

Source code checked in, #421

punzie — Tue, 31 Aug 2010 10:52:49 GMT

v0.2.0.1

Source code checked in, #420

punzie — Tue, 31 Aug 2010 10:47:53 GMT

fix IDENTITY INSERT

Updated Wiki: Home

punzie — Sat, 28 Aug 2010 11:55:07 GMT

FluentETL - Data automation made easy for coders

Automate data transfers with a few lines of code. Replace SSIS with concise yet powerful .NET code. Much simpler and easier to learn than Rhino ETL, yet flexible enough to use just about any data source.

Getting started

Here's how easy it is to copy data from one database to another (identical table schema).

using FluentETL;
using FluentETL.Impl;

...

ISource src = new OleDbSource()
                    .FromQuery("select * from Projects")
                    .Connection(oleDbConnObj);

IDestination dest = new SqlDestination()
                    .ToTable("Projects")
                    .Connection(sqlConnObj)
                    .IdentityInsert()
                    .DeleteFirst();

Transform t = new Transform(src, dest)
                    .AutoMap()
                    .Execute();

You can replace "src" and "dest" with instances of various ISource and IDestination classes.

ISource	IDestination
SqlSource	SqlDestination	For SQL Server data
OleDbSource	OleDbDestination	For OLE DB data
TextSource	TextDestination	For text files and text streams
ObjectSource	CustomDestination	For custom data sources and processing, with LINQ support

AutoMap

Transform.AutoMap() maps common field names between source and destination. If destination has no known target fields (like TextDestination), all source fields are mapped.

Reading text

ISource src = new TextSource()
                .From(@"C:\temp\sample.txt")
                .ParseAsDelimited(',', true)                          // Split w/ comma, with first line
                                                                      // as header to get field names
                .ParseAsFixedColumns(new int[] { 15, 10, 20 }, false) // Parse with fixed columns, no header
                                                                      // Field names are "Field0", "Field1", etc.
                .ParseWith(line => from s in line.Split('\t')
                                   select s.Trim(), false)            // Split with tabs, then trim; no header
                ;

Writing text

View any source data with the following code.

IDestination dest = new TextDestination()
                .To(Console.Out)
                .WithHeader(true)
                .WriteAsDelimited("\t");

Transform t = new Transform(src, dest).AutoMap().Execute();

Custom data using LINQ

Leverage LINQ for joins, aggregates, custom data sources and code-generated data.

ISource src = new ObjectSource()
                    .From(
                        from x in Enumerable.Range(1, 1000)
                        select new { Name = "Person" + x, Age = 20 + (new Random(x).Next()) % 20 }
                    );

AutoMap() works with ObjectSource -- it uses the public properties of your objects as the field names.

Transform mapping

Transform t = new Transform(src, dest)
                    .Map("Id")                                        // Map field, same name in destination
                    .Map("Manager", "ManagerName")                    // Map to different field name
                    .Map("Age", "Age", x => int.Parse(x.ToString()))  // Type conversion
                    .Map<string>("Address", "Address", x => x.Trim()) // Map then trimming the value,
                                                                      // using strongly-typed generic method
                    .Unmap("Salary")                                  // Remove mapping -- useful after doing AutoMap()

                    // Decide per row which operation to perform (Insert/Update/Delete),
                    // calls appropriate method in IDestination. Default is RowOperation.ProcessDefault
                    .DetermineOperation(row =>
                    {
                        if ((int)row["Id"] >= 100)
                            return RowOperation.Insert;
                        return RowOperation.Ignore;
                    })

                    .Execute(); // Finally, execute

Transform events

TODO

Custom insert/update/delete

For SqlDestination and OleDbDestination, using the FromTable() method presets default logic on how to insert, update and delete to your database. You may need to customize how to perform these operations, e.g. using a stored procedure. Use the InsertWithAction() or InsertWithCommand() methods to customize how to insert, for example.

TODO

More stuff

How about transactions, logging and all that? Since it's all in code, you can put FluentETL code within your own transaction code, add logging, parallelize transforms, etc. It's as powerful as you can code it!

Updated Wiki: Home

punzie — Sat, 28 Aug 2010 11:53:52 GMT

FluentETL - Data automation made easy for coders

using FluentETL;
using FluentETL.Impl;

...

ISource src = new OleDbSource()
                    .FromQuery("select * from Projects")
                    .Connection(oleDbConnObj);

IDestination dest = new SqlDestination()
                    .ToTable("Projects")
                    .Connection(sqlConnObj)
                    .IdentityInsert()
                    .DeleteFirst();

Transform t = new Transform(src, dest)
                    .AutoMap()
                    .Execute();

You can replace "src" and "dest" with instances of various ISource and IDestination classes.

ISource	IDestination
SqlSource	SqlDestination	For SQL Server data
OleDbSource	OleDbDestination	For OLE DB data
TextSource	TextDestination	For text files and text streams
ObjectSource	CustomDestination	For custom sources, with LINQ support

ISource src = new TextSource()
                .From(@"C:\temp\sample.txt")
                .ParseAsDelimited(',', true)                          // Split w/ comma, with first line
                                                                      // as header to get field names
                .ParseAsFixedColumns(new int[] { 15, 10, 20 }, false) // Parse with fixed columns, no header
                                                                      // Field names are "Field0", "Field1", etc.
                .ParseWith(line => from s in line.Split('\t')
                                   select s.Trim(), false)            // Split with tabs, then trim; no header
                ;

Writing text

View any source data with the following code.

IDestination dest = new TextDestination()
                .To(Console.Out)
                .WithHeader(true)
                .WriteAsDelimited("\t");

Transform t = new Transform(src, dest).AutoMap().Execute();

Custom data using LINQ

Leverage LINQ for joins, aggregates, custom data sources and code-generated data.

ISource src = new ObjectSource()
                    .From(
                        from x in Enumerable.Range(1, 1000)
                        select new { Name = "Person" + x, Age = 20 + (new Random(x).Next()) % 20 }
                    );

AutoMap() works with ObjectSource -- it uses the public properties of your objects as the field names.

Transform mapping

Transform t = new Transform(src, dest)
                    .Map("Id")                                        // Map field, same name in destination
                    .Map("Manager", "ManagerName")                    // Map to different field name
                    .Map("Age", "Age", x => int.Parse(x.ToString()))  // Type conversion
                    .Map<string>("Address", "Address", x => x.Trim()) // Map then trimming the value,
                                                                      // using strongly-typed generic method
                    .Unmap("Salary")                                  // Remove mapping -- useful after doing AutoMap()

                    // Decide per row which operation to perform (Insert/Update/Delete),
                    // calls appropriate method in IDestination. Default is RowOperation.ProcessDefault
                    .DetermineOperation(row =>
                    {
                        if ((int)row["Id"] >= 100)
                            return RowOperation.Insert;
                        return RowOperation.Ignore;
                    })

                    .Execute(); // Finally, execute

Updated Wiki: Home

punzie — Sat, 28 Aug 2010 11:51:05 GMT

FluentETL - Data automation made easy for coders

using FluentETL;
using FluentETL.Impl;

...

ISource src = new OleDbSource()
                    .FromQuery("select * from Projects")
                    .Connection(oleDbConnObj);

IDestination dest = new SqlDestination()
                    .ToTable("Projects")
                    .Connection(sqlConnObj)
                    .IdentityInsert()
                    .DeleteFirst();

Transform t = new Transform(src, dest)
                    .AutoMap()
                    .Execute();

You can replace "src" and "dest" with instances of various ISource and IDestination classes.

ISource	IDestination
SqlSource	SqlDestination	For SQL Server data
OleDbSource	OleDbDestination	For OLE DB data
TextSource	TextDestination	For text files and text streams
ObjectSource	CustomDestination	For custom sources, with LINQ support

ISource src = new TextSource()
                .From(@"C:\temp\sample.txt")
                .ParseAsDelimited(',', true)                          // Split w/ comma, with first line
                                                                      // as header to get field names
                .ParseAsFixedColumns(new int[] { 15, 10, 20 }, false) // Parse with fixed columns, no header
                                                                      // Field names are "Field0", "Field1", etc.
                .ParseWith(line => from s in line.Split('\t')
                                   select s.Trim(), false)            // Split with tabs, then trim; no header
                ;

Writing text

View any source data with the following code.

IDestination dest = new TextDestination()
                .To(Console.Out)
                .WithHeader(true_
                .WriteAsDelimited("\t");

Transform t = new Transform(src, dest).AutoMap().Execute();

Custom data using LINQ

Leverage LINQ for joins, aggregates, custom data sources and code-generated data.

ISource src = new ObjectSource()
                    .From(
                        from x in Enumerable.Range(1, 1000)
                        select new { Name = "Person" + x, Age = 20 + (new Random(x).Next()) % 20 }
                    );

AutoMap() works with ObjectSource -- it uses the public properties of your objects as the field names.

Transform mapping

Transform t = new Transform(src, dest)
                    .Map("Id")                                        // Map field, same name in destination
                    .Map("Manager", "ManagerName")                    // Map to different field name
                    .Map("Age", "Age", x => int.Parse(x.ToString()))  // Type conversion
                    .Map<string>("Address", "Address", x => x.Trim()) // Map then trimming the value,
                                                                      // using strongly-typed generic method
                    .Unmap("Salary")                                  // Remove mapping -- useful after doing AutoMap()

                    // Decide per row which operation to perform (Insert/Update/Delete),
                    // calls appropriate method in IDestination. Default is RowOperation.ProcessDefault
                    .DetermineOperation(row =>
                    {
                        if ((int)row["Id"] >= 100)
                            return RowOperation.Insert;
                        return RowOperation.Ignore;
                    })

                    .Execute(); // Finally, execute

Updated Wiki: Home

punzie — Sat, 28 Aug 2010 11:50:22 GMT

FluentETL - Data automation made easy for coders

using FluentETL;
using FluentETL.Impl;

...

ISource src = new OleDbSource()
                    .FromQuery("select * from Projects")
                    .Connection(oleDbConnObj);

IDestination dest = new SqlDestination()
                    .ToTable("Projects")
                    .Connection(sqlConnObj)
                    .IdentityInsert()
                    .DeleteFirst();

Transform t = new Transform(src, dest)
                    .AutoMap()
                    .Execute();

You can replace "src" and "dest" with instances of various ISource and IDestination classes.

ISource	IDestination
SqlSource	SqlDestination	For SQL Server data
OleDbSource	OleDbDestination	For OLE DB data
TextSource	TextDestination	For text files and text streams
ObjectSource	CustomDestination	For custom sources, with LINQ support

AutoMap

Transform.AutoMap() maps common field names between source and destination. If destination has no known target fields (like TextDestination), all source fields are mapped.

Reading text files

ISource src = new TextSource()
                .From(@"C:\temp\sample.txt")
                .ParseAsDelimited(',', true)                          // Split w/ comma, with first line
                                                                      // as header to get field names
                .ParseAsFixedColumns(new int[] { 15, 10, 20 }, false) // Parse with fixed columns, no header
                                                                      // Field names are "Field0", "Field1", etc.
                .ParseWith(line => from s in line.Split('\t')
                                   select s.Trim(), false)            // Split with tabs, then trim; no header
                ;

Testing/viewing data sources

View any source data with the following code.

IDestination dest = new TextDestination()
                .To(Console.Out)
                .WithHeader(true_
                .WriteAsDelimited("\t");

Transform t = new Transform(src, dest).AutoMap().Execute();

LINQ data source

Leverage LINQ for joins, aggregates, custom data sources and code-generated data.

ISource src = new ObjectSource()
                    .From(
                        from x in Enumerable.Range(1, 1000)
                        select new { Name = "Person" + x, Age = 20 + (new Random(x).Next()) % 20 }
                    );

AutoMap() works with ObjectSource -- it uses the public properties of your objects as the field names.

Transform mapping

Transform t = new Transform(src, dest)
                    .Map("Id")                                        // Map field, same name in destination
                    .Map("Manager", "ManagerName")                    // Map to different field name
                    .Map("Age", "Age", x => int.Parse(x.ToString()))  // Type conversion
                    .Map<string>("Address", "Address", x => x.Trim()) // Map then trimming the value,
                                                                      // using strongly-typed generic method
                    .Unmap("Salary")                                  // Remove mapping -- useful after doing AutoMap()

                    // Decide per row which operation to perform (Insert/Update/Delete),
                    // calls appropriate method in IDestination. Default is RowOperation.ProcessDefault
                    .DetermineOperation(row =>
                    {
                        if ((int)row["Id"] >= 100)
                            return RowOperation.Insert;
                        return RowOperation.Ignore;
                    })

                    .Execute(); // Finally, execute

Created Release: FluentETL 0.2 (Aug 28, 2010)

punzie — Sat, 28 Aug 2010 11:48:05 GMT

Complete set of basic ISource and IDestination objects
Comprehensive Transform events
Sample application

Released: FluentETL 0.2 (Aug 28, 2010)

Sat, 28 Aug 2010 11:48:05 GMT

Complete set of basic ISource and IDestination objects
Comprehensive Transform events
Sample application

Updated Wiki: Home

punzie — Sat, 28 Aug 2010 11:44:05 GMT

FluentETL - Data automation made easy for coders

using FluentETL;
using FluentETL.Impl;

...

ISource src = new OleDbSource()
                    .FromQuery("select * from Projects")
                    .Connection(oleDbConnObj);

IDestination dest = new SqlDestination()
                    .ToTable("Projects")
                    .Connection(sqlConnObj)
                    .IdentityInsert()
                    .DeleteFirst();

Transform t = new Transform(src, dest)
                    .AutoMap()
                    .Execute();

You can replace "src" and "dest" with instances of various ISource and IDestination classes.

ISource	IDestination
SqlSource	SqlDestination
SqlSource	SqlDestination
TextSource	TextDestination
ObjectSource	CustomDestination

ISource src = new TextSource()
                .From(@"C:\temp\sample.txt")
                .ParseAsDelimited(',', true)                          // Split w/ comma, with first line
                                                                      // as header to get field names
                .ParseAsFixedColumns(new int[] { 15, 10, 20 }, false) // Parse with fixed columns, no header
                                                                      // Field names are "Field0", "Field1", etc.
                .ParseWith(line => from s in line.Split('\t')
                                   select s.Trim(), false)            // Split with tabs, then trim; no header
                ;

Testing/viewing data sources

View any source data with the following code.

IDestination dest = new TextDestination()
                .To(Console.Out)
                .WithHeader(true_
                .WriteAsDelimited("\t");

Transform t = new Transform(src, dest).AutoMap().Execute();

LINQ data source

Leverage LINQ for joins, aggregates, custom data sources and code-generated data.

ISource src = new ObjectSource()
                    .From(
                        from x in Enumerable.Range(1, 1000)
                        select new { Name = "Person" + x, Age = 20 + (new Random(x).Next()) % 20 }
                    );

AutoMap() works with ObjectSource -- it uses the public properties of your objects as the field names.

Transform mapping

Transform t = new Transform(src, dest)
                    .Map("Id")                                        // Map field, same name in destination
                    .Map("Manager", "ManagerName")                    // Map to different field name
                    .Map("Age", "Age", x => int.Parse(x.ToString()))  // Type conversion
                    .Map<string>("Address", "Address", x => x.Trim()) // Map then trimming the value,
                                                                      // using strongly-typed generic method
                    .Unmap("Salary")                                  // Remove mapping -- useful after doing AutoMap()

                    // Decide per row which operation to perform (Insert/Update/Delete),
                    // calls appropriate method in IDestination. Default is RowOperation.ProcessDefault
                    .DetermineOperation(row =>
                    {
                        if ((int)row["Id"] >= 100)
                            return RowOperation.Insert;
                        return RowOperation.Ignore;
                    })

                    .Execute(); // Finally, execute

Source code checked in, #348

punzie — Sat, 28 Aug 2010 11:31:05 GMT

exclude Data from build

Source code checked in, #347

punzie — Sat, 28 Aug 2010 11:25:23 GMT

v0.2, fix OLE DB default update, add PreTransform and PostTransform to DB destinations, data refresh

Source code checked in, #345

punzie — Sat, 28 Aug 2010 09:53:32 GMT

add validation event, examples project + data, OLE DB default insert/update commands

Source code checked in, #343

punzie — Sat, 28 Aug 2010 04:03:27 GMT

fix interface errors (not checked in code), add row validation step

Source code checked in, #333

punzie — Fri, 27 Aug 2010 16:31:39 GMT

rationalize Transform events, add ISource.InTransformContext