Populate Elasticsearch Index

This Page Shows How To Populate an Elasticsearch Index using Rules

This documentation is based on the FoodCrunch use case. Please open the link below alongside this page to understand the examples.

Introduction and Use Case

If you are new to Rules, we highly recommend reading the following document before continuing:

Rules

The steps provided here are applicable to both self-hosted installations and Squidex Cloud.

Prerequisites

In order to create this rule you will need a working installation of Elasticsearch. Elasticsearch can be installed either as a Docker container, or in Kubernetes. Refer to the following links for installation instructions:

• Docker installation: https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html

• Kubernetes Installation: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-installing-eck.html

In this example, we are using the cloud version of ElasticSearch at https://elastic.co. You can subscribe to a 14 day trial of the Elasticsearch cloud here.

Squidex also provides an action for OpenSearch. The reason OpenSearch is also available is because Elasticsearch blocks all non-official drivers.

OpenSearch action has the same logic and works similarly.

Retrieving ElasticSearch Values for Squidex Rule

1. Select your Elastic deployment from the dashboard.

Select Elastic cluster

1. Copy the endpoint value next to Kibana (2). This is the Server Url value which will be required later.

Copy Kibana endpoint

1. The default user is elastic. The password is the same that has been set during Elastic cluster deployment. Make a note of them, these will also be required later in the steps. In case there is a need to reset the password, it can be done from security settings of the deployment. Select Security from the deployment and then click Reset password.

Reset elastic user password (if needed)

Creating the Elasticsearch Rule

Add a Rule

Navigate to Rules (1) in the Squidex App and select + New Rule (2).

Create a new rule

Select Content changed (3) as the trigger for this event. This will load additional settings for the trigger that can be configured such as Conditions.

Select when to trigger the rule

In this example, we are setting this rule for the magazine schema only and triggering it for all events in the schema where the event status is Published. Click + (4) to add a condition, select the schema (5) from the dropdown (i.e. magazine in this case) and use the following condition (6) in Javascript format:

event.status == 'Published'

Set the condition for the trigger

Next, select Elasticsearch (7) as the action.

Select ElasticSearch as the connector

Provide the ElasticSearch instance values here such as the Server Url, Index Name, Username, Password etc. Click Save when done.

Configure ElasticSearch in the rule

Test the Rule

In order to test the rule we can either create a new article or update an existing article. The article / content must be in Published status for the event to trigger.

On successful trigger, you will see a status message similar to the screenshot below.

Verify that the rule ran

Verifying the Indexed Data

On successful Rule execution, one can connect to (or query) the ElasticSearch cluster to validate the creation of the documents in the Index. In this example we are using the managed cluster option through https://cloud.eastic.co, hence the screenshots and the navigation instructions refer their platform. If you are using a self-hosted ElasticSearch cluster the screenshots may be different.

To view the documents, navigate and login to the Kibana endpoint (1) (you can find the same in your Elastic Cloud home page).

Copy Kibana endpoint

Post login, select Enterprise Search (2).

Access Kibana Dashboard

Next, click Indices (3) and then select the Index (4), which in this example is search-squidex.

Select the index

Select the Documents (5) tab to view the documents created through the Rule.

Select documents tab to view documents

A newly Published article appears as MagazinePublished and an Updated article will appear as MagazineUpdated.

Customizing the Document

You can choose to customize the document that is populated in ElasticSearch. We support plain Text, Script and Liquid syntax.

Below is an example of custom document that sends only the Title of the article. It is written in the Liquid syntax. You must create a valid json string, therefore it is good to use the escape formatter.

Custom Document

Additional Information

Important Information about the Workflow

The content ID from Squidex is used as the Elasticsearch document ID.

Screenshot of ElasticSearch showing document ID

Populating to ElasticSearch is an asynchronous operation. Hence, it may take a few seconds before the updated results appear on a search.

Using ElasticSearch as the Full Text Store

You can configure Squidex to use ElasticSearch as the full text store and search directly using Squidex search API. To do this set the following values:

FULLTEXT__TYPE=elastic
FULLTEXT__ELASTIC__CONFIGURATION=[CONNECTION_STRING]
FULLTEXT__ELASTIC__INDEXNAME=[NAME_OF_THE_INDEX]

For more details and other full text parameters, refer to the appsettings.json file at: https://github.com/Squidex/squidex/blob/master/backend/src/Squidex/appsettings.json#L30