Interfaces

Users

Key Functionality

Data Imports Login

Prerequisites
Overview
Connector Data Imports
Text File Data Imports
Data Imports Administration Centre Interface
Dependent Configurations
Recommendations

This document describes how data can be imported into a project on the TOTECS platform from external data sources. The platform supports importing data from a Comma Separated Value (CSV) file that can be modified with a spreadsheet file, as well as importing data from external business systems through the use of the Connector software that is integrated with a TOTECS project.

Prerequisites

Please ensure that you have read and understand the following topic links before reading on.

Overview

For each project within the TOTECS platform there is the ability to import different kinds of data. Importing data into a project can reduce the amount time that it would otherwise take for a human being to create the data from scratch using the project's Administration Centre. An example of this would be products. A business's accounting system may already have 10,000 products created within it over many years by a number staff. Rather than a person having to manually key and create these same products in the TOTECS project over several days, using the Connector software these 10,000 products can be imported into the project and created within a matter of seconds. The data imports themselves can either create new data within a project, or update existing data if matched upon.

Data Import Methods

There are 3 methods by which data can be imported into a TOTECS project. These are:

By an administrator user uploading a CSV spreadsheet file containing data.
By the Connector software obtaining data from a business system and pushing the data into the project.
By a 3rd party who have developed their own software that can push data into a project using the TOTECS platform's Application Programming Interface (API).

TOTECS Project Data Imports Diagram

From within the Data Imports Administration Centre of a TOTECS project, administrator users have the ability to import CSV spreadsheet files by uploading the file, and then having its records processed and imported. Additionally within the interface administrator users can call the Connector software to have it obtain the data from a configured business system and import the data back into the TOTECS Project using its Data API. Alternatively the connector itself can be scheduled to export data from a business system at regular intervals and push the data into a TOTECS project using the Data API. Lastly any 3rd party software can use the API to push data into a project as long as the 3rd party software conforms to the API data structures. All of these methods provide comprehensive and flexible ways to have data created, and updated within a project.

Data Import Types

When data is imported into a project, the data needs to be grouped together based on its type. For example customer account data needs to be imported separately from the product data, we call these groupings of data "Data Import Types". Different routines are written into the TOTECS platform that can import millions of records of a given data import type in one go. Each of the data import methods (that being CSV file imports, and Connector data imports) contains routines that imports the different types of data. In the section below you can find out more information about the data import types that the methods support.

Updating Existing Data Through Data Imports

When data is imported when running either a text file data import or a connector data import, the platform will check if existing data already exists. The platform will match up data already in the project based on "Key" fields of the given data type. For example products have a field called "Product Code". If a product had its Product Code field set to 12345, and a product was trying to be imported that also was going to set the Product Code to 12345, then the platform would detect this and update the data of the existing product, instead of creating a new product. Also some data import types have settings that will cause the platform to ignore updating existing records if a match is found. More information about the data import type settings can be found in the sections below.

De-activating Existing Data Through Data Imports

When Connector data imports are run there is the ability for the imports to deactivate the data that is no longer being imported. An example of this is if a product with the code 12345 was previously imported in the TOTECS project from a business system, but then a person within the business system deleted the product with the code 12345, the next time that the Products Connector data import ran it would no longer import product 12345, and instead mark the existing product in the TOTECS project as being inactive. Because the product is inactive it would no longer show to users within the Trade interface, or on any content managed web site.

The Connector data imports do not delete records from the project, they only set the data to be inactive. This is a safe guard to ensure that no data is lost if a person decided to reactivate data from an external data source again.

Connector Data Imports

The TOTECS platform supports a piece of software called the Connector, which is used to obtain data from an organisation's own business system. Requests can be made by an administrator user within a TOTECS project to ask the Connector to obtain a given type of data from the business system and import back to the project. The Connector itself can be scheduled to automatically run data imports and push data into a TOTECS project at regular intervals.The Connector provides the key component to allow a TOTECS project to integrate with another Business system over the internet.

The Connector software uses the TOTECS platform's own Data API to import data into the project. Connections between the TOTECS platform and the Connector are done through the use of the HTTP protocol, and passing Ecommerce Standards Documents with the required data.

To find out the Business Systems that the Connector can obtain data from, as well as more information about how the Connector software can be installed and configured, head to the Connector documentation centre.

Connector Data Import Types

Listed below are different Connector data import types that the TOTECS platform supports to import data into a project from the Connector software.

Data Import Type	Description
Taxcodes	Imports a list of tax code records. Each tax code contains a percentage for amount of tax it applies to a price if it is assigned to a product. Existing taxcodes in the project are matched on the Taxcode Key field.
Products	Imports a list of product records. Each record contains details of a product. Ensure that the Taxcodes data import is run before this import so that products have the correct tax applied to them. Existing products in the project are matched on the Product Key, as well as the Product Code fields.
Product Price Levels	Imports a list of price-level records. Price levels are used with product pricing to allow multiple prices to be set for a product. Existing price levels in the project are matched on the Price Level Key field.
Product Price Level Pricing	Imports a list product price records that apply prices to price levels for products. Ensure that the Products, and Product Price Levels data imports are run before this import to ensure that pricing data will be imported into the project. Otherwise some products may not have any pricing set. Existing prices in the project are matched on the Product Key, and Price Level Key fields.
Customer Accounts	Imports a list of customer account records. Each record contains details of a customer account. Ensure that the Product Price Levels data import is run before this import so that the accounts are assigned to a price level. If this does not occur then users assigned to the customer accounts that don't have a price level set will not be able to see any products or pricing. Ensure that the Sales Representatives data import is run first to ensure that customer accounts are assigned to Sales Representatives. Otherwise users assigned to a sales representative ID may not be able to order for their assigned customer accounts. Existing customer accounts in the project are matched on the Account Code, as well as the Account Key field.
Product Kits	Imports a list of products that are assigned to other products in a kit. Ensure that the products data import is run before this import to ensure that child component products will be asigned to a kit's parent product. Existing kitted products are matched on the Product Key field.
Product Flags	Imports a list of product flags, as well as a list of products assigned to each product flag. The flags allow products to be marked with an icon, highlighted and searched upon. Ensure that the Products data import is run before this import to ensure that products will be assigned to a flag. Existing product flags are matched on the Product Flag Key field. Products are matched based on the Product Key field.
Product Quantity Pricing	Imports a list product price records that apply prices to price levels for products for given quantities. This allows quantity break pricing to be set against products. Ensure that the Products, and Product Price Levels data imports are run before this import to ensure that pricing data will be imported into the project. Otherwise some products may not have any pricing set. Existing prices in the project are matched on the Product Key, and Price Level Key fields.
Product Account Pricing	Imports a list product price records that apply prices to products for given quantities of selected customer accounts. This allows contract, and special product pricing to be applied to specific customer accounts. This pricing can also be forced upon the customer account so that if there was a cheaper price-level price for the account, the platform would still choose the more expensive account price. Ensure that the Products, and Customer Accounts data imports are run before this import to ensure that pricing data will be imported into the project. Otherwise some customer accounts may not have any pricing set. Existing prices in the project are matched on the Product Key, Customer Account Key, and quantity fields.
Sales Representatives	Imports a list of sales representative records. Sales representative records are assigned to customer accounts . Existing sales representatives in the project are matched on the Salesrep Key field.
Categories	Imports a list categories, and the product mappings that denote the products that are assigned to the categories. Ensure that the Products data import is run before this import to ensure that products will be assigned to the categories. Otherwise some products may not appear within categories. Existing categories in the project are matched on the Category ID fields. Existing products in the project are matched on the Product Key field. Settings: Create New Categories: If selected then the data import is allowed to create new categories in the mirrored category tree. If it is not selected then new categories will not be created. Update Existing Category Fields: If selected then existing categories that are matched by the data import will have its fields updated. If not selected then existing categories will not have its field data updated. Update Categories Assigned To Web Category Trees: If selected then when the data import runs and an existing category is matched, and the existing category is assigned to a category tree that is marked as having categories created solely within the project (web categories), then it will allow the fields of the matched category to be updated. If not selected then it won't update the details of existing categories assigned to a web category tree. Update Products Assigned to Categories in Web Category Trees: If selected then when the data import runs and an existing category is matched, and the existing category is assigned to a category tree that is marked as having categories created solely within the project (web categories), then it will allow the products of the matched category to be updated. If not selected then it won't update/deactivate the product associations of existing categories assigned to a web category tree.
Customer Account Contracts	Imports a list of contracts records, as well as the customer account associations with contracts, and the contract associations with products. This allows users to see contracts that are associated with their assigned customer account. Note that this data import does not import the pricing associated with the contracts. That data is imported seperately in the Product Account Pricing data import. Ensure that the Products, Customer Accounts, and Product Account Pricing data imports are run first to ensure that the contracts can be viewed correctly by users. Existing contracts in the project are matched on the Contract ID field. Customer accounts are matched on the Account Key field, products are matched on the Product Key field.
Product Alternate Codes	Imports a list of codes that are associated with product records. These alternate codes can be used to allow users to search for products on codes linked to the products. Ensure that the Products data import is run first to ensure that the import can find the products to associate the codes with. Existing products in the project are matched on the Product Key field.
Locations and Product Stock	Imports a list of locations, and the associations with products and the stock level quantities for products at a given location. Ensure that the Products data import is run first to ensure that the import can find the products to associate the locations with. Existing locations in the project are matched on the Location Key field. Existing products in the project are matched on the Product Key field.
Product Attributes	Imports a list of Attribute Profiles, Attribute Fields, and attribute values associated with products. This data import allows additional data to be set against products which is grouped based on attribute profiles and field names. Ensure that the Products data import is run first to ensure that the import can find the products to associate the locations with. Settings: Create New Product Attribute Profiles: If selected then the data import is allowed to create new attribute profiles if an existing profile cannot be matched. If it is not selected then new profiles will not be created, nor will the attribute values associated with the products for profile be set. Update Existing Product Attribute Profiles: If selected then the data import is allowed to update the details of an attribute profile, as well as its attribute fields.
Product Combinations	Imports a list of Product Combination Profiles, Profile Fields, and products associated with combination based on field values. Ensure that the Products data import is run first to ensure that the import can find the products to associate with when assigning products to combinations.

Text File Data Imports

The TOTECS platform supports importing data into project from files generated in spreadsheet applications such as Microsoft Excel. This allows administrator users of a project to manage data in a spreadsheet application, then import it into a project. The exact type of spreadsheet file that can be imported is the Comma Seperated Values file format. This file format allows the data be viewed in spreadsheet applications, and also in any application that can open a text file, such as Notepad, Wordpad, or Word, since the data is stored in a human readable form as normal text (as opposed to a binary form which is cannot be understood by humans directly). For this reason the platform calls any of the CSV files that can imported as "Text File" data imports.

For all of the different Text File imports they each need to receive a CSV file that contains a header row, and subsequent data rows. The header row (which is the first row in the file) needs to contain the column names that denote where the data in the data row cells will be placed when imported. Each text file import has a list of column names that it supports, as well as columns names that are mandatory for the data import to be able to import data. This information can be found within the Data Imports interface in the Administration Centre by clicking on the "Text File Format Details" link for an text file data import.

CSV Text File Example

An example of setting up a CSV file for a "Users" text file data import would be like so:

Login	Account	Contact	User Type	Interface
GClooney	WEB_RETAIL	George Clooney	user	retail
SBullock	WEB_RETAIL	Sandra Bullock	user	wholesale

In this Users.CSV file example, the first row contains the column names that defines the kind of data that will be set for existing users or new users when the import runs. In this case when the file is uploaded and run for the Users text file data import it will create two new users for George Clooney, and Sandra Bullock (if they didn't already exist in the project with the Login IDs GClooney, SBulluck), then update the customer account, contact name, user type, and interface that the users are assigned to. There are over 30 different data fields that can be configured for the Users data import, so in the future another field labelled Email could be added the CSV file if I wanted to set the Email addresses of George Clooney and Sandra Bullock.

It is reccommended that for each CSV text file being imported that the file contains at most 10,000 data rows and is under 10MB in size. If these limits are hit then the data import may fail.

Text File Data Import Types

Listed below are different text file data import types that the TOTECS platform supports to import data into a project from CSV files.

Data Import Type	Description
Products	Imports a list of products. Each row contains details of one product. Existing products are matched on the Product Code field. Settings: Do Not Create New Products If selected then when import cannot match an existing product, it will not create a new product, otherwise if the setting is unselected it will create a new product. Continue After Unknown Products If selected then when the import cannot find a product that matches, it will stop processing any more product rows.
Product Combinations	Imports a list of product combination data. This data import allows new product combination profiles to be created, fields assigned to the combination profiles, as well as child products assigned to parent combination products based on combination field-value mappings. The import will try to find existing products by and matching on a product's code field and change them to being set as parent combination products. Each row in the CSV file allows a parent combination product to be matched to a profile, and child product that matches an existing product based on the product code field. If a child product is found then it is assigned to the parent product field the given profile field-value combination. Settings: Create Combination Profiles If selected then if the import cannot match a combination profile then it will create a new profile. If unselected then the import will not create a new profile or assign products to the profile. Create Combination Profile Fields If selected then if the import cannot find a field that matches an existing combination profile, then it will create a new field for the profile. If unselected then it will not create a new field, or allow products to be assigned to the field. Create Combination Profile Field Values If selected then then if the the import cannot find a combination field that contains the value set in the CSV file, it will create the value and assign it to the combination field. If unselected then the import will not try to create the field value, or assign products to the field or its value. Continue After Unknown Products If selected then when the import cannot find a parent product that matches, it will stop processing any more product rows in the CSV file.
Product Attributes	Imports a list of product attribute data. This data import allows new product attribute profiles to be created, as well as assign attribute values to existing products. The import will try to set attributes for products by matching products on the product code field. Settings: Create Attribute Profiles If selected then if the import cannot match on an existing attribute profile then it will create a new profile. If unselected then the import will not create a new profile or assign products to attributes in the profile. Create Attribute Profile Fields If selected then if the import cannot match on an existing attribute field name, then it will create a new attribute field. If unselected then the import will not create the profile field, or assign products to the field for any values. Merge Attributes With Product's Current Attribute Values If selected then if the import detects that a product already had attribute values assigned to it, it will then add the attribute values in the CSV text file to the product. If unselected then when the import finds a product with attribute values already assigned to it, it will clear the previous values, and set the new values contained in the CSV text file. Continue After Unknown Products If selected then when the import cannot find a product that matches, it will stop processing any more rows in the CSV file.
Categories	Imports a list of categories. The data import allows new categories to be created, as well as existing categories to be updated. Existing categories are matched on the Category Code field. Settings: Create New Categories If selected then if the import cannot match on an existing category then it will create a new category. If unselected then the import will not create a new category. Create Category Trees If selected then if the import cannot match on an existing category tree, it will then create a new category tree. If unselected then the import will not create a new category tree, or assign categories to it.
Category Products	Imports a list of product associations with categories. This import allows existing products to be assigned to existing categories. Existing products are matched on the Product Code field, existing categories are matched in the Category Code field. Settings: Continue After Unkown Products If selected then when the import cannot find a product that matches, it will stop processing any more rows in the CSV text file.
Users	Imports a list of users. The data import allows new users to be created, and existing users to have their data updated. Existing users are matched in the Login ID field. Ensure that user rows have the Account ID assigned to an existing customer account that lives in the project. Settings: Do Not Update Existing Users If selected then when the import finds a matching user, it will not update the user's details. If unselected then it will allow the details of an existing user to be updated.
Customer Accounts	Imports a list of customer accounts. The data import allows new customer accounts to be created, as well as allowing existing accounts to have their data updated. Existing customer accounts are matched on the Account ID field.
Related Products	Imports a list of product associated with other products. The import allows products to be assigned to other products in the related products data structure. Existing products are matched on the Product Code field.
Product Pricing	Imports a list of prices for a product. Prices can be set for different price levels, as well as based on specific quantity breaks. Existing products are matched on the Product Code field. Existing price levels are matched on the Price Level ID field.
Contracts	Imports a list of contracts. The data import allows new contracts to be added to the project, as well as existing contracts to be updated. When contracts are created they will be set as web contracts, that is contracts that are not imported from any other business system. Existing contracts are matched on the Contract ID field.
Contract Products	Imports a list of product associations with contracts. The import allows products to become assigned to contracts and have a price set for the contract. Existing contracts are matched on the Contract ID field. Existing products are matched on the Product Code field.
Contract Customer Accounts	Imports a list of customer account associations with contracts. This import allows customer accounts to become assigned to contracts. Existing contracts are matched on the Contract ID field. Existing customer accounts are matched on the Account ID field.

Data Imports Administration Centre Interface

Within the Administration Centre of a TOTECS project, under the Data menu, there is a menu item labelled Data Imports that takes an administrator user to the interface. The Data Imports interface allows administrators to import data into a TOTECS project by uploading CSV text files, or by running Connector data imports that obtain data from an external business system that is connected via the Connector software..

The following actions below can be done within the Data Imports Administration Centre interface.

Import Data From A CSV Spreadsheet Text File

To import data from a CSV text file for a given data imoprt, follow these steps:

Open a web browser and navigate the Administration Centre for the relevant TOTECS project.
Click on the Data menu button.
Click on the Data Imports menu item link.
For the relevant import type row click on the Modify Settings link if it exists. Tick any options in the Import Settings dialog then click the Close button
Click on the Upload File button in the row of the relevant data import.
In the File Browser window locate the file on the filesystem and click the browsers Open button (or however your operating system labels the button in the file browser). The file will then be uploaded to the server and being processing the data.

Once the Progress bar has changed to 100% aginst the data import the data will have been successfully imported. Click on the History and Error Log tabs to view the details of the data import running. Go to the relevent part of the Administration Centre to check that the correct data was imported into the TOTECS project.

Import Data From The Connector

To import data from the Connector for a given data imoprt, follow these steps:

Open a web browser and navigate the Administration Centre for the relevant TOTECS project.
Click on the Data menu button.
Click on the Data Imports menu item link.
Click on the Connector Imports tab.
Click on the tick box in the Selected column next to each of the data imports that you wish to run. Untick the data imports that you do not wish to run. The data imports interface will remember the imports that were previously selected.
For the selected data imports if in the Modify Settings column it displays the Modify Settings link, click on the link to modify any settings that will customise how the given data import will run. Close the dialog once you have made your choices. Note that the settings will be remembered for the next time the data import is run within the Administration Centre.
Click on the Start Import(s) button. The platform will run each data import sequentially. In the Progress column you will see the progress bar change as the data is being imported for each import type.

Once that data import(s) have completed you can then view the amount of records that imported by clicking on the History tab. If you click on the Error Log tab then you can see any errors that may have occurred during the data import process.

Additionally with the data imported you can then navigate to the section of the Administration Centre that shows the relevant data to see the data that was imported.

View Data Imports History

To view the history of data imports that have run previously to import data into a TOTECS project, follow these steps:

Open a web browser and navigate the Administration Centre for the relevant TOTECS project.
Click on the Data menu button.
Click on the Data Imports menu item link.
Click on the History tab.

The history tab will display a table of the last 100 data imports that were run. Additionally it will display the amount of records that were imported at the time. Note that this history will show both Connector Data Imports run from both the Administration Centre, or from the Connector itself, as well as Text File Data Imports.

View Data Import Errors

To view the errors that occurred when a data import previously tried import data into a TOTECS project, follow these steps:

Open a web browser and navigate the Administration Centre for the relevant TOTECS project.
Click on the Data menu button.
Click on the Data Imports menu item link.
Click on the Error Log tab.

The Error Log tab will display a list of errors that occurred when data imports previously run. Use the error log as a guide to see the cause of why data imports may fail, or do not import some records in a data import.

Dependent Configurations

To allow administrator users to see the Data Imports interface they must be assigned to an Administrator Permission Role that has the "View Data Imports" permission set to Allow.
To allow administrator users to import data using the Data Imports interface they must be assigned to an Administrator Permission Role that has the "Modify Data With Data Import and API" permission set to Allow.
The file size CSV text files that are uploaded must be under 10MB in size.

To run the Connector data imports within the Data Imports Administration Centre the Connecter software must be installed on a computer, have an adaptor setup within it to connect to a business system, then within the Administration Centre under the Data menu, in the Connector settings interface the settings need to be configured so that the TOTECS platform can communicate with the Connector's adaptor.

Setting	Description
Connector Location	Public IP address or domain to connect to the computer that the Connector software is installed on.
Connector Port	Port that the Connector software is listening on.
Adaptor Key	The key of the adaptor configured within the Connector software to use to connect with the external business system.

Recommendations

WIthin the adaptor of the Connector software that is configured to the TOTECS platform, data exports can be scheduled to export data from a business system and import the data into the TOTECS platform at regular times of the day, week, or month.
WIthin the adaptor of the Connector software that is configured to the TOTECS platform, configure each data export to specify to fields that will be updated when the data is exported from the business system and imported in the TOTECS project.
Within the Search Users Administration Centre interface administrator users can export existing user data to a CSV spreadsheet file. Use this spreadsheet file to update the details of users and then import the file back in via the Users Text File data import. The header column names in the exported file are set to match the Users data import column names to make it easy to import the file in again.
Within the Search Products Administration Centre interface administrator users can export existing product data to a CSV spreadsheet file. Use this spreadsheet file to update the details of products and then import the file back in via the Products Text File data import. The header column names in the exported file are set to match the Products data import column names to make it easy to import the file in again.
Before running any data import it is recommended to check any configurable settings related to a given data import, since these settings can affect how the data import is run and the data that it can create or update. Data imports can overwrite existing data, so it is important to understand the kinds of data that is being imported, and the fields that the data will be placed into.
For example, an administrator user may set the first description field for each product through the TOTECS Administration Centre's Product Editor, but then they run a Products connector data import which is configured to set the first description field to come from the product name field in their business system. Because the data import was run it has overwritten the existing product description data that was originally entered by the administrator. This scenario could have been avoided if the Products data export in the Connector software was configured to place the product name data into a different field. The take home message is be careful on how the imports are configured, notify any administrator people who manage data about how the configurations and processes have been setup, and try to manage data from one location as much as possible.

Document last modified on the 4th June 2015 by Annonymous