Association Rules Model for problem

What would be the right design approach for the following problem?

I have a single table called SelectionFactors, which has the following columns and sample data:

ProjectID Factor FactorValue 1000 Countries USA 1000 Countries Canada 1000 Countries France 1000 Languages English 1000 Languages French 1000 Company Type Consulting 1000 Company Type Software 2000 Countries India 2000 Countries China 2000 Countries USA 2000 Languages English 2000 Languages Chinese (Simplified) 2000 Languages Chinese (Traditional) 2000 Languages Spanish 2000 Company Type Retail 2000 Company Type Dairy Products

The problem is to allow a descriptive analysis of the data to find patterns in the users selections. For instance,

if Languages->English is selected, what are the counts of projects for other Factor->Factor Value combinations?

Countries->USA = 2, Countries->Canada=1, Company Type->Consulting=1 and so on.

Since all the data is in this single table, are both the case and nested tables the same? What are the keys and inputs? I only need a descriptive analysis (no prediction) and ALL possible combinations MUST be part of the results; how should the model be designed?

Thank you,

Anna.

This is a good problem. What you want to do is to create a model with three nested tables all marked predict and input. Event though you want descriptive analysis, you need to mark them "predict" such that rules will be formed (only items marked predict will show on the right hand side of rules).

The way you will create such a model is to use the "Named Query" feature of the data source view(DSV). You can right-click on the DSV and select "New Named Query" to create these.

The named queries you will want to create are these:

Projects: SELECT DISTINCT ProjectID FROM SelectionFactors

Companies: SELECT ProjectID, FactorValue FROM SelectionFactors WHERE Factor='Companies'

Languages: SELECT ProjectID, FactorValue FROM SelectionFactors WHERE Factor='Languages'

Company Type: SELECT ProjectID, FactorValue FROM SelectionFactors WHERE Factor='Company Type'

You will then need to relate the ProjectID in each of the transaction named queries (Companies, etc) to the ProjectID in the Projects named query. The system will prompt you to set ProjectID as the key of the Projects named query . At this point, the named queries are equivalent to tables as far as Analysis Services is concerned.

When you create the model, you will make the Projects table the case table and the other tables nested tables. ProjectID will be the key of the Projects table and FactorValue will be the key of the other tables. Make sure to mark FactorValue as Key, Input, and Output.

When you process your model you will see rules relating projects, companies, and company types (both between factos and in the same factor).

Note that you may need to adjust the MINIMUM_SUPPORT and MINIMUM_PROBABILITY parameters to get the results you need (in fact, it's recommended)

HTH

-Jamie

|||

Thanks for your response.

However, I should have mentioned that there is an open-ended number of "Factors". New factors are added often and the number of factors is nearly 100.

The type of questions I am trying to answer using this model is:

"Among projects that have Languages-> English, how many (distinct projects) have Company Type->Consulting, Countries->USA" and so on.

Ideally, I would like to provide this analysis in a free, OLAP environment where the user can select one factor-> factor value combination and then see all the other related combinations.

I currently have a Cube with SelectionFactors serving as both the Fact and the dimension table with a single measure - DISTINCT COUNT of projectIDs.

Thanks once again.

|||

Given the information in my previous post, what are my design options?

Thanks.

|||

My question for you, then, is if you have a cube for this, what are you not getting that you need?

If the factor/factorvalue can be viewed as a complete unit, and you don't need to provide analysis between factors - e.g. given Language X, which Countries and Company Types are prevalent, you could create a calculated column in the DSV that combines the factor and factor value.

|||

Perhaps you can help me understand why I am unable to get the expected results. I am fairly new to OLAP and Data Mining and would appreciate any pointers and guidance.

The full picture of my problem is:

Dimension Project Profile - Project ID, (Other Project related attributes)

Dimension Selection Factors - Factor, Factor Value

Fact Selection Factors - Project ID, Factor, Factor Value

Project Cube Measure - DISTINCT COUNT of Project IDs

My understanding is that this is a problem of intersection of factors, which I cannot achieve with a single dimension. In fact there is this other thread http://forums.microsoft.com/msdn/showpost.aspx?postid=856304&siteid=1 that seems to come very close to my problem.

In my case, the following MDX returns the projects that have the intersection of specific factor values.

SELECT {[Measures].[Project Count]} ON COLUMNS,

INTERSECT

(

NONEMPTY

(

[Project Profile].[Project ID].Children,

(

[Measures].[Project Count],

[Selection Factor].[Selection Factors].[Factor].&[Annual Revenue].&[$500 million - $3 billion in revenues]

)

),

NONEMPTY

(

[Project Profile].[Project ID].Children,

(

[Measures].[Selection Project Count],

[Selection Factor].[Selection Factors].[Factor].&[Concurrent Users].&[1,000–10,000 users]

)

)

)

ON ROWS

FROM PROJECTCUBE

The questions are:

1. How do I get all the other Factor->Factor Values that are part of the selection factors for the projects returned by this MDX. This, if I understand correctly, is Basket analysis.

and

2. Is there a generic way to perform this analysis through cube design rather than writing MDX for each case?

Thanks once more.|||

The following MDX gives me the contents of the "baskets" of projects that have the selected items. However, the counts of the projects is not restricted to the subset retrieved through the Intersect. Is there a way to get the counts to only be within the subset returned by the Intersect?

With this MDX, I can get very close to what I need. The only thing I am missing is to get the counts (perhaps through a calculated measure). Help on this will be much appreciated!

__

SELECT [Measures].[Project Count] ON COLUMNS,

NONEMPTY

(

([Selection Factor].[Factor].Children, [Selection Factor].[Factor Value].Children),

INTERSECT

(

NONEMPTY

(

[Project Profile].[Project ID].Children,

(

[Measures].[Selection Project Count],

[Selection Factor].[Selection Factors].[Factor].&[Annual Revenue].&[$500 million - $3 billion in revenues]

)

),

NONEMPTY

(

[Project Profile].[Project ID].Children,

(

[Measures].[Selection Project Count],

[Selection Factor].[Selection Factors].[Factor].&[Concurrent Users].&[1,000–10,000 users]

)

)

)

)

ON ROWS

FROM PROJECTCUBE

|||

In order to obtain counts restricted to the subset returned by the Intersect, this MDX takes the constraint out of the ROWS into the WHERE clause.

I would still like to know how to design an Association rules model that can "automate" the generation of results by case. Hopefully, the MDX can point directly to the expected results.

SELECT [Measures].[Project Count] ON COLUMNS,

NONEMPTYCROSSJOIN([Selection Factor].[Factor].Children, [Selection Factor].[Factor Value].Children)

ON ROWS

FROM PROJECTCUBE

WHERE

NONEMPTY(

INTERSECT

(

NONEMPTY

(

[Project Profile].[Project ID].Children,

(

[Measures].[Project Count],

[Selection Factor].[Selection Factors].[Factor].&[Annual Revenue].&[$500 million - $3 billion in revenues]

)

),

NONEMPTY

(

[Project Profile].[Project ID].Children,

(

[Measures].[Project Count],

[Selection Factor].[Selection Factors].[Factor].&[Concurrent Users].&[1,000–10,000 users]

)

)

)

)

sql

Association Questions

I'm wondering if anyone can give me some help with an association model I'd like to setup. It's a typical market-basket analysis, but rather than grouping by individual customers, I'd like to group by customer grouping. (In our database, customers are grouped into categories like: large, small, medium) If this is possible, I'd like to generate the most popular items (so just querying the most probable itemsets), for each customer grouping (I'll refer to this as 'segments' from here on out), and then create a listing of customers in each segment which do not have the most popular items for their segment. I know for this last part I can use reporting services to tackle that problem, however, I'm not really sure how I can really do the rest of this with an association model in SSAS.

Our table structure looks like this:

Code Snippet

CustomerTable PurchasesTable

- --

CustomerName(key) CustomerName

CustomerGroup PurchasedProduct

And the data is arranged in this fashion:

Code Snippet

Customer Table:

CustomerName CustomerGroup

- -

A large

B large

C small

Purchases Table:

CustomerName PurchasedProduct

-

A ProductA

A ProductB

B ProductA

C ProductC

C ProductD

I know this is a lot of information but any help you guys may be able to offer would be great! Thanks!

Just a suggestion which might not do exactly what you requested, but might help:

You could create one association model looking like below:

(

Customer TEXT KEY,

CustomerGroup TEXT DISCRETE, // i.e. used as input, on the left hand side of the rules

Products TABLE PREDICT

(

Product TEXT KEY

)

)

Then, for each customer, get the recommended association predictions based on:

- the products currently owned by the customer

- the group that the customer belongs to

This will create rules that might include the customer group on the left hand side (if that it found to be interesting). It may also generate rules which ignore the group, containing items that sell together across groups of customers.

In the prediction query, if you ask solely for predictions that are NOT based on rules (but on popular itemsets), you will get a list of popular itemsets that should match each customer according to previous purchases and possibly group. Of course, excluding predictions that are based on rules might actually reduce the accuracy of the recommendations. You might want to sort the prediction results by support and use some sort of threshold

If you must really separate the groups (i.e. require to take the customer group into account for each individual customer), you could also build 3 models, one for each group. Then, you can use the same kind of predictions, but a different model for each customer (based on the customer group). This way, the recommendations will be based solely on popular items within the current customer group.

Hope this helps

|||

I was thinking that might be the way to go about doing that, but just have a couple quick questions to clarify.

If I do it the way you suggested, you said that I might get rules that ignore the group/contain the group. How may I filter(?) the results so I only obtain those with with the groups, as I'm not as interested in what individual customers purchased, so much as just purchases made within the groups?

If I built three models, how would I tell the model to only be concerned with the 'small' group when all the groups belong in the same database? Is there somewhere I can specify a filter on the mining model to only focus on certain fields...or something?

Thanks for your help so far!

|||

In SQL Server 2005 you would need to build three separate views (or named queries in the DSV) each showing only a single group of customers. Then for each group you build a separate mining structure and a separate model.

We are working on some advances that may make this easier in future versions, but for now, you should find that this approach works well.

Association Prediction by Rules Still Returns Itemsets

If I use this code with an association model, it still returns itemsets for me - when it should be returning only nodes with rules associated with them (according to sqlserverdatamining.com). If I try adding 'AND $PROBABILITY > .25' to the where clause, it returns 0 results for every query I try. Any clue why this may be happening?

Code Snippet

SELECT FLATTENED
(SELECT * FROM PredictAssociation([Product],20,
INCLUDE_NODE_ID,INCLUDE_STATISTICS)
WHERE $NODEID<>'')
FROM
[ProductRecommend]
PREDICTION JOIN
OPENQUERY([ds],
'SELECT
[PRODUCTCLASSID],[DESCRIPTION]
FROM
[Product_Table]
WHERE
[PRODUCTCLASSID] = ''1234'' AND [DESCRIPTION] = ''DESC''
') AS t

ON
[ProductRecommend].[Product].[PRODUCTCLASSID] = t.[PRODUCTCLASSID] AND
[ProductRecommend].[Product].[DESCRIPTION] = t.[DESCRIPTION]

This query returns more relevant results than those lacking the filtering by $NODEID, however the results should have higher probabilities than .047! Please help! Thanks!

Okay, I just reconstructed the same query using my data in a relational mining model (instead of OLAP) and got reasonable results. Would anyone know how to fix this for OLAP or be able to point me in a direction where I could go to learn how to do it? Thanks.

|||What is the MINIMUM_PROBABILITY value for the OLAP mining model?
Is it different than the one from the relational model?

One more thing (which you probably know already) -- the query you posted initially executes one prediction (PredictAssociation ... 20) for each row in the data source query (does not group together multiple input rows belonging to the same transaction).
Basically, only rules having a single item on the left hand side will be used in prediction.

Association model prediction not using itemsets

I have a market basket model using associations. It generated several dozen itemsets. However when I attempt to run a singleton prediction like this:

select (Predict(Orderproduct3q,INCLUDE_STATISTICS,10)) as [Recommendation]

From

[Case All]

NATURAL PREDICTION JOIN

(SELECT (SELECT '16407' AS [Pname])) AS t1

the resulting predictions don't take the itemsets into account. Instead, the predictions consist of the ranked products in the training set, ordered by frequency. This appears to happen regardless of the precise query specified within the "natural prediction join".

What's going on here and how do I generate a singleton prediction which makes use of the itemsets?

The nested table inside the input should have the same name as the nested table of the model, for Natural prediction Join. Could you please try this:

select (Predict(Orderproduct3q,INCLUDE_STATISTICS,10)) as [Recommendation]

From

[Case All]

NATURAL PREDICTION JOIN

(SELECT (SELECT '16407' AS [Pname]) AS Orderproduct3q ) AS t1

Otherwise, the input rowset (t1) contains an unnamed nested table which cannot be mapped to your model's table

|||

Thanks for the suggestion, which was on the right lines. Something like this does now pick up one of the itemsets.

select (Predict(Orderproduct3q,INCLUDE_STATISTICS,10)) as [Recommendation]

From

[Case All]

NATURAL PREDICTION JOIN
(SELECT (SELECT '17717' AS Pname Union SELECT '16415' AS Pname) as Orderproduct3q ) AS t1

|||Also look at the tip/trick at http://www.sqlserverdatamining.com/DMCommunity/TipsNTricks/3514.aspxsql

Association Mining Model Predictions

Hi,

I've been playing around with the association mining model in SQL server 2005 and built a market-basket analysis of my data that I'm pretty happy with. The next task for me is figuring out how to run DMX queries against the data that I've just mined, so we may possibly use it in a web based application. This wouldn't necessarily be a difficult problem (and still may not be), but every example I've seen for the Mining Model Prediction Designer uses relational databases and I built my mining model off OLAP. Therefore, my predictable attribute is nested and when relating the mining model structure to the relational database that the cube was built off always gives me an error:

"Errors in the high-level relational engine. The 'CompanyName' column could not be found in the top-level clause of the SHAPE statement."

What I would like to do, and I'm not really even sure how I should structure any of my queries, is feed the model a product and have it return a listing of all the products it predicts. Currently, I've only been able to get the designer mode to process a singleton query, and even that didn't return any useful data. I know that this probably can be done pretty easily so any advice you may be able to offer would be greatly appreciated!!

So you may better understand my question, my association mining structure hierarchy looks as this..

[Model] ProductRecommend

[Case][Key]CustomerList

[Case][Attribute]CompanyName

[NestedTable]Product

[Nested][Key]PRODUCTCLASSID

[Nested][Attribute]PRODUCT

With that in mind, I'm trying to perform a query simliar to this:

SELECT

PredictProbability([ProductRecommend].[Product].[PRODUCTCLASSID]), <- Throws Error for PredictProbability syntax no matter what I try to get to [PRODUCTCLASSID]

(SELECT [PRODUCT] FROM [ProductRecommend].[Product])

From

[ProductRecommend]

NATURAL PREDICTION JOIN

(SELECT 'test' AS [COMPANYNAME],

(SELECT '1234' AS [PRODUCTCLASSID],

'ProductA' AS [PRODUCT]) AS [Product]) AS t

Thanks again for any help!

Just incase someone out there runs into the same trouble I had, I think I've made some progress. My mistake was that I was using the PredictProbability function when I should have been using the PredictAssociation function. PredictAssociation does exactly what I wanted to do with the query, now I just need to figure out how to structure the query so it's simply based off an input product.|||What do you mean "simply based off an input product"?|||

Well it looks like 'simply based off an input product' isn't so simple... Since I've finally gotten queries to run, they've all pretty much returned the same associations. I feel like I've been everywhere on the internet looking for how to do what I described above, but just can't find enough information to solve my problem. My overall goal is to create a recommendation system similiar to Amazon.com. I'm pretty sure my mining model achieves this goal, but writing the DMX do so is a little bit difficult based on the structure of our database, and the fact that I modeled the Mining Structure off our OLAP cube.

Basically I just want to say:

SELECT

Predict Association([Product],10)

FROM

[ProductRecommendation]

NATURAL PREDICTION JOIN

(SELECT (SELECT 'ProductCustomerHasSelected' AS [ProductName], 'ProductKey' AS [PRODUCTCLASSID]) AS [Product])

And have it return:

ProductCustomerHasSelected -- Predicts Product A, B, D, and E

Is this the wrong way to go about doing that?

|||

BTW, have you looked at Raman Iyer and Jesper Lind's article? Create a Web Cross-sell Application

It has code examples, including how to construct the queries. You can see the app they build in action on a little webcast I did at.

In this article they use the following query:

SELECT FLATTENED

TopCount(Predict([Customer Movies], INCLUDE_STATISTICS), $AdjustedProbability, 5)

FROM [Movie Recommendations]

NATURAL PREDICTION JOIN

( SELECT ( SELECT 'Star Wars' AS [Movie] UNION SELECT 'The Matrix' AS [Movie] ) AS [Customer Movies] ) AS t

This returns the top 5 movies associated with the input movies: Star Wars and the Matrix in this case.

There is an excellent explanation of the thinking behind this query in the article.

hth

Association Browser Error

Hello Developers,

I used the add mining model to mining structure to modify a model so that maximum itemset =2, min prob=.01, min support= 2.

When i select maximum rows to anything higher than 2000 (default) i get duplicate rules.

The maximum rules returns is exactly16000 even though i set it higher than that.

Any ideas on the causes?

Thanks

Davy

Can you post the version/build you're using?|||

I am using the newest version of the Excel 2007 Data mining Add-in

I'm thinking that the rules are duplicated because I built a new model into the structure with a lower min_sup and lower min_prob, than the original model in the structure. I will test my hypothesis and let you know how it goes.

|||The duplicates appeared when I selected any Maximum Row level higher than 2000. The duplicates disappeared after I made another selection after that.|||

This may be a bug. Can you get in touch with Microsoft Support so we can investigate?

( Re. you previous post: The viewer behavior should not be affected by the presence of other models in the mining structure. )

Association Browser Error

Hello Developers,

I used the add mining model to mining structure to modify a model so that maximum itemset =2, min prob=.01, min support= 2.

When i select maximum rows to anything higher than 2000 (default) i get duplicate rules.

The maximum rules returns is exactly16000 even though i set it higher than that.

Any ideas on the causes?

Thanks

Davy

Can you post the version/build you're using?|||

I am using the newest version of the Excel 2007 Data mining Add-in

I'm thinking that the rules are duplicated because I built a new model into the structure with a lower min_sup and lower min_prob, than the original model in the structure. I will test my hypothesis and let you know how it goes.

|||The duplicates appeared when I selected any Maximum Row level higher than 2000. The duplicates disappeared after I made another selection after that.|||

This may be a bug. Can you get in touch with Microsoft Support so we can investigate?

( Re. you previous post: The viewer behavior should not be affected by the presence of other models in the mining structure. )

Association Browser Error

Hello Developers,

I used the add mining model to mining structure to modify a model so that maximum itemset =2, min prob=.01, min support= 2.

When i select maximum rows to anything higher than 2000 (default) i get duplicate rules.

The maximum rules returns is exactly16000 even though i set it higher than that.

Any ideas on the causes?

Thanks

Davy

Can you post the version/build you're using?|||

I am using the newest version of the Excel 2007 Data mining Add-in

I'm thinking that the rules are duplicated because I built a new model into the structure with a lower min_sup and lower min_prob, than the original model in the structure. I will test my hypothesis and let you know how it goes.

|||The duplicates appeared when I selected any Maximum Row level higher than 2000. The duplicates disappeared after I made another selection after that.|||

This may be a bug. Can you get in touch with Microsoft Support so we can investigate?

( Re. you previous post: The viewer behavior should not be affected by the presence of other models in the mining structure. )

Association algorithm itemsets

What is the algorithm that generates the itemsets in the Association model? I'm looking to possibly use this part of the Association algorithm (i.e. the grouping into itemsets) in a separate plug-in algorithm.

The algorithm is based on the 'a priori' technique. Basically, 1-itemsets are filtered based on certain thresholds, then the algorithm moves to computing 2-itemsets and so on. Brief details on how Microsoft Association Rules work: http://msdn2.microsoft.com/en-us/library/ms174916.aspx

Hope this helps

Assertion failed error while resizing a vector in a plug-in algorithm

I got an Assertion failed error while resizing a vector in a plug-in algorithm.

In order to isolate the problem I created a simple model class in Navigator.h file as shown below:

//========================= begin code =======================

class CStateStats
{
public:
DOUBLE m_dblSum;
DOUBLE m_dblSqrSum;
public:
CStateStats()
{
m_dblSum = 0.0;
m_dblSqrSum = 0.0;
}
};

class CAttStats : public DMHALLOC
{
public:
dmh_vector<CStateStats> vstatestats;
public:
CAttStats() : vstatestats (*this)
{
}
};

//========================= end code =======================

The access to DMHALLOC is provided in that class and in Navigator class as shown below:

//========================= begin code =======================

class ATL_NO_VTABLE NAVIGATOR :

public DMHALLOC,

public CComObjectRootEx<CComMultiThreadModel>,

public CComCoClass<NAVIGATOR, &CLSID_NAVIGATOR>,

public ISupportErrorInfo,

public IDMAlgorithmNavigation

{

public:

NAVIGATOR() : _viAttributeOutput(*this), _vCAttStats(*this)

//========================= end code =======================

I succeded making room for _vCAttStats vector, but when I tried providing room for the vectors of the vector I got an Assertion failed error (file dmhallocator.h Line:56 Expression assert(_dmhalloc._spidmmemoryallocator != NULL)). Please, see the code below, included in NAVIGATOR::GetNodeArrayProperty function:

//========================= begin code =======================

_vCAttStats.resize (2); // <<<<< succeeded here!

// make space for the states

_vCAttStats[0].vstatestats.resize(ulStates); // <<<<<<< assertion failed here!

//========================= end code =======================

I tried using a vector-of-vector approach and I also succeeded.

But I have to use that kind of structure: a vector of class with a vector inside.

I think I must provide a similar approach of vector-of-vector existing in DmhVector.h but I don't know how to do it.

I would apreciate any help.

hello,Claudio

It looks like:

- the original DMHALLOC (containing the allocator pointer), NAVIGATOR is passed properly to the vector of CAttStats, _vCAttStats and used correctly in resizing it

- during resize, STL calls the default constructor for CAttStats , which does not take any argument. Consequently, the CAttStats DMHALLOC instance does not contain an allocator pointer

- _vCAttStats[0].vstatestats.resize(ulStates); attempts to resize the vstatestats vector, which does not have a properly initialized DMHALLOC , hence the assertion that fails

To confirm my hypothesis in your code, try to use _vCAttStats[0].vstatestats.get_allocator() right before calling resize. It is likely NULL or uninitialized (well, not the actual object which is a reference, but the IDMMemoryAllocator pointer it contains).

The simplest and most reliable solution is:

- do not derive CAttStats from DMHALLOC

- use, instead, an IDMMemoryAllocator member inside the CAttStats

- explicitly set that member to the current allocator before using vstatestats

- use IDMMemoryAllocator->Alloc( n*sizeof(CStateStats) ) to get a C-style buffer of state stats which can be freed in the destructor of CAttStats with IDMMemoryAllocator->Free

Hope this helps

|||

Hi Bogdan,

Thanks for you help.

I suceeded using your suggestions.

The drawbacks of this approach are that I have to explicitly initialize vstatestats because the class constructor is not called anymore when I provide room for the buffer and I don't have anymore the vector facilities. Am I correct?