IMP Central Data Repository
The Intelligent Message Processor Central Data Repository (IMP-CDR) Platform is a central data consolidation point for clusters of IMPs deployed with the service provider. For these clusters, the IMP-CDR acts as the repository for all shared information where it generates a consolidated metadata index for each object visible within the domain, IP gateway address or user. IMP-CDR accounts for both external reputation sources and any relevant reputation-related information gathered from the deployed IMP cluster itself.
The IMPs in the deployed clusters can use this consolidated reputation index to filter traffic while minimizing outbound queries. This is supported by using the IMP-CDR as a caching/replication server for third-party content filters. Administrators can use the reputation data for domain reputation for sender authentication, which is necessary for the full application of DKIM or SPIF on the incoming IMPs. To maximize filtering performance, operators can access the available shared data in the IMP-CDR through complex rules from individual IMPs in the clusters, providing a shared brain for the clusters.
Benefits
- Minimize outbound and inbound traffic by aggregating and consolidating external and internal reputation data
- Validate statistical data collected by individual IMPs in a cluster by comparing it with other internal and external data
- Facilitate complex statistical post-processing and filters acting on statistics gathered across the IMP cluster and externally
- Detect abnormal traffic patterns (such as attacks) through automated analysis of correlated and aggregated data from IMP clusters
- Provide policies for controlled sharing of data with third parties (other operators or reputation providers) — only available in V2.0 and later
- Improve cluster-filtering performance by using consolidated reputation data.
- Provide domain reputation for sender authentication (DKIM, SPIF)
- Provide centralized management of E-mail certification services (token handling for Goodmail, for example)
- Provide centralized caching/replication system of data for third-party applications (application and partner-dependent) and reputation data
Features
The IMP-CDR Platform is a special purpose deployment of the IMP. Its purpose is to provide a near real-time shared brain facility to facilitate complex correlation of data internal to the cluster and external data. This maximizes filtering capabilities, attack detection and options for complex filtering rules and minimizes network queries to outside systems. The system typically deploys in a cluster of two systems - at a minimum - for redundancy and availability.
Individual IMPs operating in a cluster analyze the characteristics of the traffic flow through the IMP by user, domain and IP gateway addresses to gather relevant reputation data sending E-mails through the system. Operators can use these characteristics in dynamic rules to identify problematic senders, gateways and domains and activate configured counter-measures to deal with the problems.
It is becoming difficult to detect increasingly sophisticated attacks with data derived from the message traffic flow through just one system. To counter these intelligent attacks requires directed and policy-driven sharing of reputation data and any other information within IMP's single-use clusters and amongst other clusters (either internally or externally). Through complex near real-time analysis and correlation of this data, the IMP-CDR can recognize second-order effects from sophisticated attacks. To counter these attacks, the IMPs can receive partial raw data or the IMP clusters can use an aggregated index with generated data.
The collected and consolidated data - gathered from the internal clusters accessible to the IMP-CDR as well as any external reputation data - is configurable and available to the IMPs deployed in the internal clusters. The external reputation data ranges from commercial or public DNS-based blacklists and whitelists to URL lists. The data can use the verdict of content filters and anti-virus systems on streams of E-mails received from users, domains or gateways to modify the reputation of those entities.
The IMP-CDR supports multiple policies to handle aggregation and consolidation of the reputation data. Different policies can manage the availability of the data to other systems such as the IMPs in the internal clusters or other data users. These same policies assign reputations for queries regarding DKIM or SPIF-authenticated E-mails, providing a holistic view of the reputation of the sending system.
The policies used in the IMP-CDR for aggregating and consolidating the incoming information streams can execute complex rules. This ensures reputation information is directly visible from the data streams and allows for the building up of derived reputation.
For example, derived reputation data might come from comparing the number of virus-infected E-mails sent from a particular gateway or user, the ratio of virus-infected E-mails to clean E-mails from that gateway or user and the change in ratios over time.
It is possible to combine any available information relative to these message streams indexed by IP gateway address, user, domain or similar data with other relevant data to generate useful comparison information. The correlated information can help with making decisions on what to do with specific messages or flows by the IMPs in internal clusters. Customers can also share the information openly or in a closed environment with other IMP-CDRs or other systems.
Based on customer and deployment requirements, an IMP deployment may need to reference several external information sources. These range from simple DNS-based blacklists or whitelists to complex, proprietary queries for content filters or E-mail authentication tokens. To minimize the load on the provider and the customer using these services, the IMP-CDR has the capability to support central caching, replication or both for most major reputation providers.
Caching can be a simple replication of the data or it can combine with policies and rules to consolidate data in flexible, customer-driven structures. Dependent on the third-party application and its integration with the IMP, the platform supports various replication capabilities.
As with all IMP policies, any IMP-CDR rules and policies outlined in this document can be entered through the Workflow Policy Processor UI present in the IMP-CDR or through the similar UI of the IMP-CM Platform. It is also just as easy to modify the policies in the UI. The user-friendly, intuitive structure of the workflow policies provides flexibility in managing information and its flow throughout the cluster and beyond. This set-up simplifies implementation even for complex rules.
The IMP-CDR platform supports administration and screen level privileges allowing various read, write and visibility access for different user types such as operations, customer support, and billing and provisioning management. The platform supports full audit trails for individual users. Where available, it is possible to include integration into the IMP-CM Platform.
The IMP-CDR platform provides for all the automated access capabilities that the individual IMP has integrated. Therefore, access to the consolidated data is available by using its REST/XML/SOAP infrastructure from any other system requiring the use of that data.
System Requirements
The IMP-CM Platform requires each managed IMP in a cluster to operate IMP firmware OOS version 2.5 or greater with the following modules at a minimum:
- IMP Client IMP-CDR Module
The IMP-CDR Platform cannot be used on E-series IMPs or with clusters of E-series IMPs. To deploy redundant platforms using the IMP-CDR Platform clusters, use IMP O550s at a minimum.
The IMP-CDR Platform itself should be operated on a cluster of two IMP O550s or more.
IMP Central Data Repository (IMP-CDR) Shared Brain