The CSV importer can be used to import and update large amounts of data into the CollectionPro system. This can include simple lists, hierarchical thesauri, or image and object data.
You can create the data for example in Microsoft Excel, Apple Numbers, or Notepad++. If you’re working with Excel or Numbers, you must save the data in .csv format with UTF-8 or UTF-16 encoding before importing. The .xls, .xlsx, or .numbers file types are not supported.
General information
- File encoding: The CSV file must be encoded in either UTF-8 or UTF-16 format. This ensures that letters like ä, ö and special characters are imported correctly.
- Column separator: Columns can be separated by a comma, semicolon, or tab. CollectionPro automatically detects the separator.
- Text blocks: Text that contains commas, semicolons, tabs, or line breaks should be enclosed in quotation marks. CollectionPro automatically detects this.
- Column headers: The CSV file must contain column headers. You can choose the headers freely, but you will need to manually map the source and destination fields (see options).
- Automatic field mapping: If you use CollectionPro’s internal field names as column headers in the CSV file, the fields are mapped automatically. The easiest way to get the built-in field names is to first download a record in CSV format and copy the column headers from it.
- Unique identifier: Always use a unique identifier where possible. Only this identifier can be used to update existing datasets later.
Please note the following examples when importing text with quotation marks:
| Text to be imported | Text in the CSV file |
|---|---|
| “Plain text enclosed in quotes to be imported.” | “””Plain text enclosed in quotes to be imported.””” |
| Simple text that contains “quotation marks”. | “Simple text that contains “”quotation marks””.” |
| “Plain text in quotation marks, but which also contains “quotation marks”.” | “””Plain text in quotation marks, but which also contains “”quotation marks””.””” |
| “This is a quote.” said Person Z. | “””This is a quote.”” said Person Z.” |
| Person Z said “This quote.” | “Person Z said “”This quote.””” |
Settings
The CSV importer offers various settings and options, which are explained below. Please also read the general information before importing data with the CSV importer.
Import settings
After uploading a file in the “CSV file” field, the following options are available:
| Option | Description |
|---|---|
| First click on this button to select a CSV file for the import. Via the “X” you can remove the file again. | |
| CSV field names | Select the row where the column names are located so that they are not imported as records. |
| Target field names | Select the row that contains the target field names so the import mapping is done automatically and they are not imported. |
| Object type | Select the object type that you want to import the data into. |
| Pool | Select the pool in which the new records should be created. The pool is set only when records are inserted. If existing records are updated via the CSV importer, the pool will not be changed. For this purpose, please use the group editor. |
| Mask | Select the mask to use for the import. |
| File upload method | Select a method for uploading files. Direct: The file is downloaded and then uploaded using /eas/put. URL (Remote PUT): The file is not downloaded and /eas/put is accessed directly from the file URL. The file is downloaded from the server and uploaded. (This option is the fastest). Ignore files: All files are ignored and not imported. |
| Metadata mapping | Select the metadata mapping to use when importing files. |
| Tag mode | Select whether tags from the CSV file should be added to existing tags or should replace existing ones when updating records. |
| Field for updates | Select a field here to identify existing records and update them with the contents from the CSV file. |
| Append records in nested fields | Select whether to add the contents of the CSV file to existing nested fields. |
| Append records in reverse nested fields | Select whether to add the contents of the CSV file to existing reverse nested fields. |
| Create linked records | Specify whether linked records should be created during the actual import or not. |
| Pool for linked records | Set up different pools for the linked records. |
| Comment | Enter a comment for the CSV import, which will appear in the change history of the imported / updated records. |
| Chunk size | Size of the processing chunks that are sent to the server. In case of very complex data models and data volumes, a timeout may occur. In this case, try using a smaller chunk size. |
| Show display names | Choose whether to show display names or internal field names for CollectionPro fields. |
Overview
Below the import settings, an import overview is displayed after preparing. The checkboxes are used to filter the entries in the table view.
Import mapping
Fields for which no mapping has been selected will not be changed when updating records that already exist in the system. Fields for which a mapping has been selected, but for which there is no content in the CSV file, will be emptied when updating records.
Table view
In the table view, you can see the data from your CSV file. After preparing and importing, additional information is automatically added (recognizable by the column names “fylr|”). For more information about this, see the “Protocol” section at the end of this article.
Record Preview
In the “Record Preview” tab, you can see the data that would be imported according to the import mapping. Please note that this is only a preview of the mapped fields. When updating already existing records the result may differ.
JSON Preview
In the “JSON preview” tab, you can see the data that would be imported according to the import mapping in the JSON structure.
Options
The lower part of the CSV importer contains the following functions:
| Button | Description |
|---|---|
| Reload | Reloads the CSV file and discards any information already loaded by preparing it. |
| Download CSV | When preparing and after saving, more information is generated, which is written back to your CSV file (see table below). With “Download CSV” you can save this file to your desktop. |
| Download import settings | Download the configured import settings and the import mapping as a JSON file. |
| Upload import settings | Upload the JSON file with the import settings and import mapping. |
| Prepare | Prepares the CSV import. This includes checking the data, as well as searching for existing and linked records. |
| Insert | Starts the actual CSV import and adds any new records. Already existing records will not be updated. |
| Update | Starts the actual CSV import and updates the existing records. No new records are created. Note however, that empty columns in the CSV will cause the update to empty the contents of these fields in the records. |
| Insert & update | Performs both inserts and updates directly in sequence. |
Protocol
After preparing and after importing, the system generates certain information. These can be seen in the table view of the CSV importer or downloaded as a CSV file after the import.
| Row | Description |
|---|---|
row_idx | Number of the row, starting with 0. |
| operation | Action that will be performed (“insert”, “update”, “delete”). |
| status | Import status (“ready”, “invalid”, “failed”, “done”). Rows with status “invalid” cannot be imported (example: wrong date format). Rows with the status “failed” were not imported correctly (example: mandatory field violation). |
| timestamp | The date + time when the CSV import/update/deletion occurred. |
| status_text | Contains further information about the error that occurred in the case of the status “invalid” and “failed”. |
| warning_text | Contains more information in case of trying to import invalid JSON into custom data type fields. |
| ID | ID of the record found in CollectionPro (for records created by CSV import, this column will be blank). |
| version | Version of the record. |
| id_parent | ID of the parent entry (for hierarchical object types only). |
| depth | Depth of the record (for hierarchical object types only), starting with 0. |
| path | The path of the record (for hierarchical object types only). |
| eas_ids|file | Contains the internal EAS ID of the file uploaded using the CSV importer (see Upload files). |
| eas_ids|file|metadata | Contains metadata taken from the file uploaded using the CSV importer (see Upload files). |
| object type|mask | For all linked object types a separate column is generated. After preparing you can see whether the entries referred to already existed (an ID is displayed for them), whether they are newly created (they appear with “new”) or, if the option “Create linked records” is deactivated, whether they were not found (they appear with “searching”). |