Centers

Addressing difficult data problems

3forge Centers add features that address data problems software engineers routinely encounter.

• Real-time streaming analytics
• Replication, sharding and web load balancing without additional code
• Streaming inserts to historical tables with immediate consistency
• Historical table sparse updates
• Advanced conflation
• Self-discovery schemas
• Remote procedure calls
• Testing and code coverage tools
• Query planner
• Multi-threaded queries
• Historical schema changes without backfilling data

In-memory cache SQL columnar database

Every source a Center touches resolves into one live picture of the business, held in memory and queried in SQL. Applications stop interpreting streams and start reading state.

Columnar and in memory means the view a desk asks for has already been computed by the time the question arrives, rather than assembled once it does.

One SQL dialect spans both stores, so a query never has to know whether the rows it wants are still in memory or already on disk.

Historical petabyte SQL columnar database

While the highest query performance is achieved with real-time databases, 3forge also offers columnar historical tables capable of holding trillions of rows. Persisted to disk and supporting partitioning, these tables are designed for storing large volumes of data at high speed with fast retrieval, all while using the same SQL syntax as real-time tables.

Data from historical tables can be queried and loaded into a real-time table where the full breadth of querying optimizations can be accessed, including joins with other tables.

• Large column counts and heavy data types such as blob fields
• Schema management. Add, drop or modify columns without impacting historical partitions. Partition columns are immutable, so careful planning matters when designing the table schema. The HDB maps older partitions to new schemas seamlessly, preserving historical integrity
• Row-level operations. UPDATE and DELETE clauses are supported while preserving partition optimization. Significant changes within a partition are re-optimized automatically, and sort indexing further improves query performance

The system dynamically adapts storage types during optimization on a partition basis, for disk efficiency and query speed based on actual data usage. Each column can be configured for optimal performance:

Complex event processor

3forge offers real-time tables and database-native functionality that allow business logic to be executed on the fly based on data updates. Database-native triggers can aggregate, filter, join, enrich and transform information as source tables change, and custom logic can run in response to inserts, updates and deletions.

Optimizations for large loads

3forge Centers use several advanced approaches to balance the need for speed in data transmission with the technical constraints that could otherwise slow the data flow.

Archiving real-time data

Streaming updates move from real-time tables into the historical database using event, batch or timer-driven approaches, so historical records stay current without interrupting ongoing operations.

Nothing is exported, staged or reloaded. Today's ticks become last year's history in place, which is why a single query can read now and then at once.

Complex event processing

Six trigger types ship with the platform, and they are the whole vocabulary of the event processor:

• Aggregation to create and update aggregation tables
• Projection for filtered tables
• Join to create joined tables
• Decorate to enrich the data with additional data from another table
• Relay for sending messages through the 3forge Relay
• AmiScript to run custom scripts upon any insert, update or delete activity

Triggers are declared in SQL, so the event logic lives with the data rather than in a service beside it that has to be deployed, versioned and kept in step.

-- A script trigger
CREATE TRIGGER <trigger_name> OFTYPE AMISCRIPT ON
  <table_name> [PRIORITY priority] USE...

-- An aggregation trigger
CREATE TRIGGER <trigger_name> OFTYPE AGGREGATE ON
  <source_table>,<target_table> USE...

-- A projection trigger
CREATE TRIGGER <trigger_name> OFTYPE PROJECTION ON
  <source_tables>,<target_table> USE...

The trigger types at work

A real-time P&L grid is not one calculation but a chain of them, each trigger doing one job as the data changes beneath it:

• 3forge joins the drop-copy of fills with instrument data to normalize it
• 3forge filters fills to requested symbols and aggregates intraday considerations
• 3forge stores inbound market data for symbols and FX rates, filters to retain needed prices, and projects to a dedicated table
• 3forge combines start-of-day with intraday P&L and projects to the grid
• 3forge decorates the grid data with updated prices and FX rates