FluentETL - Data automation made easy for coders

Automate data transfers with a few lines of code. Replace SSIS with concise yet powerful .NET code. Much simpler and easier to learn than Rhino ETL, yet flexible enough to use just about any data source.

Getting started

Here's how easy it is to copy data from one database to another (identical table schema).

using FluentETL;
using FluentETL.Impl;

...

ISource src = new OleDbSource()
                    .FromQuery("select * from Projects")
                    .Connection(oleDbConnObj);

IDestination dest = new SqlDestination()
                    .ToTable("Projects")
                    .Connection(sqlConnObj)
                    .IdentityInsert()
                    .DeleteFirst();

Transform t = new Transform(src, dest)
                    .AutoMap()
                    .Execute();

You can replace "src" and "dest" with instances of various ISource and IDestination classes.

ISource IDestination
SqlSource SqlDestination For SQL Server data
OleDbSource OleDbDestination For OLE DB data
TextSource TextDestination For text files and text streams
ObjectSource CustomDestination For custom data sources and processing, with LINQ support


AutoMap

Transform.AutoMap() maps common field names between source and destination. If destination has no known target fields (like TextDestination), all source fields are mapped.

Reading text

ISource src = new TextSource()
                .From(@"C:\temp\sample.txt")
                .ParseAsDelimited(',', true)                          // Split w/ comma, with first line
                                                                      // as header to get field names
                .ParseAsFixedColumns(new int[] { 15, 10, 20 }, false) // Parse with fixed columns, no header
                                                                      // Field names are "Field0", "Field1", etc.
                .ParseWith(line => from s in line.Split('\t')
                                   select s.Trim(), false)            // Split with tabs, then trim; no header
                ;

Writing text

View any source data with the following code.

IDestination dest = new TextDestination()
                .To(Console.Out)
                .WithHeader(true)
                .WriteAsDelimited("\t");

Transform t = new Transform(src, dest).AutoMap().Execute();

Custom data using LINQ

Leverage LINQ for joins, aggregates, custom data sources and code-generated data.

ISource src = new ObjectSource()
                    .From(
                        from x in Enumerable.Range(1, 1000)
                        select new { Name = "Person" + x, Age = 20 + (new Random(x).Next()) % 20 }
                    );

AutoMap() works with ObjectSource -- it uses the public properties of your objects as the field names.

Transform mapping

Transform t = new Transform(src, dest)
                    .Map("Id")                                        // Map field, same name in destination
                    .Map("Manager", "ManagerName")                    // Map to different field name
                    .Map("Age", "Age", x => int.Parse(x.ToString()))  // Type conversion
                    .Map<string>("Address", "Address", x => x.Trim()) // Map then trimming the value,
                                                                      // using strongly-typed generic method
                    .Unmap("Salary")                                  // Remove mapping -- useful after doing AutoMap()

                    // Decide per row which operation to perform (Insert/Update/Delete),
                    // calls appropriate method in IDestination. Default is RowOperation.ProcessDefault
                    .DetermineOperation(row =>
                    {
                        if ((int)row["Id"] >= 100)
                            return RowOperation.Insert;
                        return RowOperation.Ignore;
                    })

                    .Execute(); // Finally, execute

Transform events

TODO

Custom insert/update/delete

For SqlDestination and OleDbDestination, using the FromTable() method presets default logic on how to insert, update and delete to your database. You may need to customize how to perform these operations, e.g. using a stored procedure. Use the InsertWithAction() or InsertWithCommand() methods to customize how to insert, for example.

TODO

More stuff

How about transactions, logging and all that? Since it's all in code, you can put FluentETL code within your own transaction code, add logging, parallelize transforms, etc. It's as powerful as you can code it!

Last edited Aug 28, 2010 at 11:55 AM by punzie, version 14