Skip to main content

Development Environment

Click Data Intelligence Development > Development Environment in the left navigation to enter the development environment management interface, where you can view the list of development tasks under the current workspace. From this page, you can create, edit, start/stop development tasks, view logs, and enter the task IDE interface.

Create Development Task​

Click the Create Development Task button to open the task configuration page. Fill in the required fields including name, resource configuration (CPU, memory, storage size), and description. Click Create to create a new development task.

Under resource configuration, set a custom Code-server image or leave it blank to use the platform default development image. GPU count requests NVIDIA GPUs for the development environment and defaults to 0. These resources and the image are separate from those used by published Ray tasks. Storage capacity is for the development environment persistent volume; allow space for code, Conda environments, and dependencies.

View Details, Edit, Delete, Start/Stop, and Logs​

Each development task record in the list includes View Details, Edit, Delete, Start/Stop, View Logs, and Enter IDE buttons on the right side for managing the current development task.

Enter the Task IDE Editor​

For a running development task, click the Go to Develop button to enter the code development interface, where you can write Flink SQL, Spark SQL, and Python code, and publish completed code as approval tasks.

Upon entering the task IDE interface, select the /home/coder/work-dir directory for code editing.

Create Flink/Spark SQL and Publish for Approval​

Right-click in the left tree panel to create a Flink/Spark SQL note named user-demo.sql. Once created, you can edit Flink/Spark SQL and execute the SQL statements.

  • Click Run on activate connection and select the lakesoul-flink-global or lakesoul-spark-global connection in the SQL Tool Connection dialog to execute the current SQL.

  • Select the code to execute, right-click, and choose Run Selected Query, then select the lakesoul-flink-global or lakesoul-spark-global connection to execute the selected SQL.

  • After execution, the results of the current SQL or selected SQL are displayed in the right-side output panel.

  • Click the Task Configuration button in the top right corner to publish Flink streaming/batch or Spark batch SQL for approval. After publishing, it can run in streaming or batch mode with scheduled execution.

In the task configuration panel, choose whether to create a new task or update an existing launched task, select the corresponding Flink/Spark SQL file for SQL content, choose the Flink or Spark processing engine (and for Flink, set streaming or batch mode), configure resource settings, and click Publish to submit it to the approval task interface in Task Management.

When developing SQL statements, you can use ${scheduleTime} to obtain the current task's schedule time timestamp.

Usage: ${scheduleTime}

Example:

  1. Add ${scheduleTime} in SQL
insert into `lakesoul`.`test_cdc`.`test` values (8, FROM_UNIXTIME(${scheduleTime}/1000,'yyyy-MM-dd HH:mm:ss'), 'test-8', 'aa-8');

Create Python Tasks and Publish for Approval​

Right-click in the left tree panel to create a Python task named user-note-1.py or user-note-1.ipynb. Once created, you can develop Python code within it.

  • Run edited Python code

    • Create a Python runtime environment

      The IDE has built-in conda support. Create an isolated Python virtual environment for dependency management:

      conda create -n lakesoul_venv python=3.10

      conda activate lakesoul_venv

      conda install -y -n lakesoul_venv -c conda-forge conda-pack

    • After creating the environment, select the newly created runtime environment from the environment selector in the bottom right corner.

Select the runtime environment, then click the Run Python File button in the top right to execute the code and view the output.

- Run Python code in Jupyter mode

    Click Select Kernel in the top right and choose the corresponding Python environment from the selection list.

Click the Run ALL button or the Execute Cell button on individual cells to run the corresponding code.

- Publish completed Python code as an approval task for scheduled execution after approval.

    1. When publishing for approval, package the current virtual environment using `conda pack -f -n lakesoul_venv -o lakesoul_venv.tar.gz` into a tar file, and upload it to the package file management via the upload package function. If the launched Python task encounters missing dependencies at runtime, use `conda install` to install the required libraries.

    2. Click the **Task Configuration** button in the top right to configure publishing for Python-type tasks.

    In the task configuration panel, choose whether to create a new task or update an existing launched task, select the Python file content, choose the Python processing engine, select the Python runtime dependency environment (you can choose an already uploaded dependency file or click Upload Package to upload a new packaged tar file), set resource settings, and click Publish to submit it to the approval task interface in Task Management.

When developing Python batch scheduling tasks, use --scheduleTime to obtain the current task's schedule timestamp. Usage is as follows:

import argparse

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="PySpark Batch Task")
    parser.add_argument("--scheduleTime", type=str, required=True, help="Set scheduler Time")

    args = parser.parse_args()
    currentDate = int(args.scheduleTime)
    previousDate = datetime.fromtimestamp(currentDate / 1000, tz=pytz.timezone("Asia/Shanghai")) - timedelta(days=1)
    preDate = previousDate.strftime("%Y%m%d")
    print(preDate)

Create Ray Job Tasks and Publish for Approval​

Create a Python file under /home/coder/work-dir in the left resource manager, and write Ray, Daft, and LakeSoul related code.

Click the Python runtime environment at the bottom right and select the default /home/coder/conda/envs/lakesoul_venv/bin/python runtime environment.

Click the Run button in the top right, select Run Python File to execute the current Python file and view the output in the terminal.

After code verification, click the Task Configuration button in the top right, select the Ray Job task type, and complete the configuration.

Ray Job publishing configuration

Select a cluster​

  • RayJob-managed: RayJob creates a cluster for the task with its own Head/Workers. Configure resources and the image in the task form. Without an explicit cleanup policy, the platform automatically releases the cluster 1800 seconds (30 minutes) after the job finishes.
  • Standalone: Select a ready, unsuspended cluster in the current workspace. Multiple tasks can share it. The task form displays the cluster's current Head/Worker, GPU, and autoscaling settings as read-only values. Change these settings on the cluster management page; task settings do not override the cluster's resources or image.

See Ray Cluster Management to create and manage Standalone clusters. A binding includes both the cluster name and UID. If a cluster is deleted and recreated with the same name, select it again in the task form. When switching back to a RayJob-managed cluster, review the RayJob-managed cluster's resources, image, and cleanup settings.

Resources and images​

  • Worker CPU / Worker Memory: Resources allocated to each Worker Pod.
  • GPU / Worker: NVIDIA GPUs per Worker; 0 requests no GPU. The platform currently uses nvidia.com/gpu; this field does not request AMD GPUs. GPU workloads still need CPU and memory.
  • Worker Autoscaling: When disabled, the configured Worker count is fixed. When enabled, a RayJob-managed cluster starts with zero Workers and has a minimum of zero, scaling with workload demand up to the configured maximum. Worker counts exclude the Head.
  • Head CPU / Head Memory: Resources allocated to the Head Pod. Start with at least 4G of Head memory.
  • Ray Image: An explicitly specified RayJob-managed cluster image takes precedence. When blank, a positive GPU / Worker count selects the platform's default GPU image; otherwise the default CPU image is used. Administrators configure these through compute_engine.ray_gpu_runtime and compute_engine.ray_runtime. Submission fails if the required default is missing. Standalone clusters use their own image.

Custom GPU images need compatible Ray, CUDA, and application dependencies. Cluster nodes must have allocatable NVIDIA GPUs; setting a GPU count does not install drivers. Ray code must also declare GPU requirements for its tasks or Actors.

Project files and runtime dependencies​

  • Ray Working Directory: Optional package type 7, accepting .zip files for project modules and resources. The platform combines the package with the task entry code to generate a per-run working_dir. An entry package is generated even when no working directory package is selected.
  • Python Archives: Optional Python environment package (type 6), accepting .zip or .tar.gz. It provides a packaged Python interpreter and dependencies, separately from project files. The archive root must contain bin/python and bin/activate, and the environment must be compatible with the cluster's OS, architecture, and Ray/Python versions.
  • Ray Runtime Env Configuration: Both cluster modes support task-level dependency and environment-variable YAML. Do not set working_dir: the platform generates it. The platform processes excludes when assembling the working directory package instead of forwarding it to Ray. The generated entry file cannot be excluded.

Without Python Archives, additional dependencies can be configured as follows:

pip:
  - pendulum==3.0.0
env_vars:
  MY_ENV: test
excludes:
  - .venv/
  - __pycache__/
  - "*.log"

When Python Archives is selected, the task uses the Python environment inside the archive. The platform currently does not support combining it with Runtime Env fields pip, conda, uv, py_executable, container, or image_uri to configure another environment. You can still set business environment variables through env_vars and exclude project files through excludes. This restriction applies only when Python Archives is selected.

See Package Management for packaging instructions.

RayJob configuration and cleanup policies​

Enter supported RayJob fields here, not a complete Kubernetes manifest. Do not include apiVersion, kind, metadata, or spec. Field names are case-sensitive. Supported fields are activeDeadlineSeconds, preRunningDeadlineSeconds, backoffLimit, submitterConfig.backoffLimit, shutdownAfterJobFinishes, ttlSecondsAfterFinished, and deletionStrategy.deletionRules.

For a Standalone cluster:

  • Omit shutdownAfterJobFinishes or set it to false.
  • Do not set ttlSecondsAfterFinished, even to 0.
  • Do not set deletionStrategy, including rules using DeleteSelf, DeleteWorkers, or DeleteNone.
  • Omit top-level backoffLimit or set it to 0. Rerun the task through the platform when needed.

For example:

activeDeadlineSeconds: 3600
preRunningDeadlineSeconds: 600
backoffLimit: 0
shutdownAfterJobFinishes: false

For a RayJob-managed cluster, choose only one of the following cleanup configurations. When all cleanup fields are omitted, the platform adds shutdownAfterJobFinishes: true and ttlSecondsAfterFinished: 1800. These defaults also apply when only unrelated settings such as deadlines or retries are provided. If any cleanup field is explicitly set, the platform uses the supplied configuration without adding cleanup defaults. To disable automatic cleanup, set only shutdownAfterJobFinishes: false.

Option 1: Release cluster resources 1800 seconds after the job finishes (the platform default).

shutdownAfterJobFinishes: true
ttlSecondsAfterFinished: 1800

A positive ttlSecondsAfterFinished requires shutdownAfterJobFinishes: true.

Explicitly setting only shutdownAfterJobFinishes: true uses KubeRay's default TTL of 0, with no additional delay after completion. When deletionStrategy is provided, the platform does not inject the defaults above. Defaults apply when submitting a new RayJob; existing RayJob resources are not modified.

Option 2: Rule-based cleanup. The platform only accepts a nonempty deletionRules list, not legacy onSuccess / onFailure fields. The installed KubeRay CRD and Operator must support this field, with RayJobDeletionPolicy enabled. See the official KubeRay documentation for policy semantics:

policyResources removed
DeleteClusterRayCluster and its Pods
DeleteWorkersWorker Pods
DeleteSelfRayJob and its associated resources, including its managed cluster
DeleteNoneNo deletion
deletionStrategy:
  deletionRules:
    - policy: DeleteCluster
      condition:
        jobStatus: SUCCEEDED
        ttlSeconds: 800
    - policy: DeleteCluster
      condition:
        jobStatus: FAILED
        ttlSeconds: 3600

condition.ttlSeconds is the nonnegative delay in seconds after that rule's terminal condition is reached. Each rule must specify exactly one of jobStatus (SUCCEEDED or FAILED) and jobDeploymentStatus: Failed. When deletionStrategy is present, omit both top-level shutdownAfterJobFinishes and ttlSecondsAfterFinished, even values of false or 0.

Publishing and editing forms display configuration conflicts. The backend also validates settings on save and submission. After switching cluster modes, remove fields that the new mode does not allow.

Click Publish to submit the task to Production Tasks > Publish Approval. After approval, the task can go live and run according to its batch scheduling configuration. Use --scheduleTime to obtain the schedule timestamp, as with Python tasks.