GitHub - PacktPublishing/LLM-Engineers-Handbook: The LLM's practical guide: From the fundamentals to...
来源: https://github.com/PacktPublishing/LLM-Engineers-Handbook
作者: None
原始日期: None
分类: coding
标签: llm, [], prompt, rag, inference, vector
English
:root { --tab-size-preference: 4; } pre, code { tab-size: var(--tab-size-preference); } {"locale":"en","featureFlags":["actions_custom_images_storage_billing_ui_visibility","actions_image_version_event","actions_service_container_command","agent_conflict_resolution","alternate_user_config_repo","arianotify_comprehensive_migration","batch_suggested_changes","billing_discount_threshold_notification","code_scanning_alert_tracking_links_phase_2","code_scanning_dfa_degraded_experience_notice","codespaces_prebuild_region_target_update","codespaces_tab_react","coding_agent_model_selection","coding_agent_model_selection_all_skus","coding_agent_third_party_model_ui","comment_viewer_copy_raw_markdown","contentful_primer_code_blocks","copilot_agent_image_upload","copilot_agent_snippy","copilot_api_agentic_issue_marshal_yaml","copilot_ask_mode_dropdown","copilot_chat_attach_multiple_images","copilot_chat_clear_model_selection_for_default_change","copilot_chat_enable_tool_call_logs","copilot_chat_explain_error_user_model","copilot_chat_file_redirect","copilot_chat_input_commands","copilot_chat_opening_thread_switch","copilot_chat_reduce_quota_checks","copilot_chat_search_bar_redirect","copilot_chat_selection_attachments","copilot_chat_vision_in_claude","copilot_chat_vision_preview_gate","copilot_custom_copilots","copilot_custom_copilots_feature_preview","copilot_diff_explain_conversation_intent","copilot_diff_reference_context","copilot_duplicate_thread","copilot_extensions_hide_in_dotcom_chat","copilot_extensions_removal_on_marketplace","copilot_features_sql_server_logo","copilot_file_block_ref_matching","copilot_ftp_hyperspace_upgrade_prompt","copilot_icebreakers_experiment_dashboard","copilot_icebreakers_experiment_hyperspace","copilot_immersive_code_block_transition_wrap","copilot_immersive_embedded","copilot_immersive_file_block_transition_open","copilot_immersive_file_preview_keep_mounted","copilot_immersive_job_result_preview","copilot_immersive_layout_routes","copilot_immersive_structured_model_picker","copilot_immersive_task_hyperlinking","copilot_immersive_task_within_chat_thread","copilot_mc_cli_resume_any_users_task","copilot_mission_control_always_send_integration_id","copilot_mission_control_cli_resume_with_task_id","copilot_mission_control_initial_data_spinner","copilot_mission_control_lazy_load_pr_data","copilot_mission_control_scroll_to_bottom_button","copilot_mission_control_task_alive_updates","copilot_org_policy_page_focus_mode","copilot_redirect_header_button_to_agents","copilot_resource_panel","copilot_scroll_preview_tabs","copilot_share_active_subthread","copilot_spaces_ga","copilot_spaces_individual_policies_ga","copilot_spaces_pagination","copilot_spark_empty_state","copilot_spark_handle_nil_friendly_name","copilot_swe_agent_hide_model_picker_if_only_auto","copilot_swe_agent_pr_comment_model_picker","copilot_swe_agent_use_subagents","copilot_task_api_github_rest_style","copilot_unconfigured_is_inherited","copilot_usage_metrics_ga","copilot_workbench_slim_line_top_tabs","custom_instructions_file_references","dashboard_indexeddb_caching","dashboard_lists_max_age_filter","dashboard_universe_2025_feedback_dialog","flex_cta_groups_mvp","global_nav_react","hyperspace_2025_logged_out_batch_1","hyperspace_2025_logged_out_batch_2","hyperspace_2025_logged_out_batch_3","ipm_global_transactional_message_agents","ipm_global_transactional_message_copilot","ipm_global_transactional_message_issues","ipm_global_transactional_message_prs","ipm_global_transactional_message_repos","ipm_global_transactional_message_spaces","issue_cca_modal_open","issue_cca_visualization","issue_fields_global_search","issues_expanded_file_types","issues_lazy_load_comment_box_suggestions","issues_react_bots_timeline_pagination","issues_react_chrome_container_query_fix","issues_react_prohibit_title_fallback","issues_react_relay_cache_index","issues_react_timeline_side_panel","issues_search_type_gql","landing_pages_ninetailed","landing_pages_web_vitals_tracking","lifecycle_label_name_updates","low_quality_classifier","marketing_pages_search_explore_provider","memex_default_issue_create_repository","memex_live_update_hovercard","memex_mwl_filter_field_delimiter","memex_remove_deprecated_type_issue","merge_status_header_feedback","notifications_menu_defer_labels","oauth_authorize_clickjacking_protection","octocaptcha_origin_optimization","primer_react_overlay_max_height_clamp_to_viewport","primer_react_spinner_synchronize_animations","prs_conversations_react","rules_insights_filter_bar_created","sample_network_conn_type","secret_scanning_pattern_alerts_link","session_logs_ungroup_reasoning_text","site_features_copilot_universe","site_homepage_collaborate_video","spark_prompt_secret_scanning","spark_server_connection_status","suppress_automated_browser_vitals","viewscreen_sandbox","webp_support","workbench_store_readonly"],"copilotApiOverrideUrl":"https://api.githubcopilot.com"} GitHub - PacktPublishing/LLM-Engineers-Handbook: The LLM's practical guide: From the fundamentals to deploying advanced LLM and RAG apps to AWS using LLMOps best practices · GitHub Skip to content {"props":{"docsUrl":"https://docs.github.com/get-started/accessibility/keyboard-shortcuts"}} /* Override primer focus outline color for marketing header dropdown links for better contrast */ [data-color-mode="light"] .HeaderMenu-dropdown-link:focus-visible, [data-color-mode="light"] .HeaderMenu-trailing-link a:focus-visible { outline-color: var(--color-accent-fg); } Navigation Menu Toggle navigation Sign in Appearance settings {"props":{}} {"props":{"should_use_dotcom_links":true}} PlatformAI CODE CREATIONGitHub CopilotWrite better code with AIGitHub SparkBuild and deploy intelligent appsGitHub ModelsManage and compare promptsMCP RegistryNewIntegrate external toolsDEVELOPER WORKFLOWSActionsAutomate any workflowCodespacesInstant dev environmentsIssuesPlan and track workCode ReviewManage code changesAPPLICATION SECURITYGitHub Advanced SecurityFind and fix vulnerabilitiesCode securitySecure your code as you buildSecret protectionStop leaks before they startEXPLOREWhy GitHubDocumentationBlogChangelogMarketplaceView all featuresSolutionsBY COMPANY SIZEEnterprisesSmall and medium teamsStartupsNonprofitsBY USE CASEApp ModernizationDevSecOpsDevOpsCI/CDView all use casesBY INDUSTRYHealthcareFinancial servicesManufacturingGovernmentView all industriesView all solutionsResourcesEXPLORE BY TOPICAISoftware DevelopmentDevOpsSecurityView all topicsEXPLORE BY TYPECustomer storiesEvents & webinarsEbooks & reportsBusiness insightsGitHub SkillsSUPPORT & SERVICESDocumentationCustomer supportCommunity forumTrust centerPartnersView all resourcesOpen SourceCOMMUNITYGitHub SponsorsFund open source developersPROGRAMSSecurity LabMaintainer CommunityAcceleratorGitHub StarsArchive ProgramREPOSITORIESTopicsTrendingCollectionsEnterpriseENTERPRISE SOLUTIONSEnterprise platformAI-powered developer platformAVAILABLE ADD-ONSGitHub Advanced SecurityEnterprise-grade security featuresCopilot for BusinessEnterprise-grade AI featuresPremium SupportEnterprise-grade 24/7 supportPricing{"resolvedServerColorMode":"day"} Search or jump to... Search code, repositories, users, issues, pull requests... --> Search Clear Search syntax tips Provide feedback --> We read every piece of feedback, and take your input very seriously. Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly --> Name Query To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up Appearance settings {"props":{}} Resetting focus You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert {{ message }} PacktPublishing / LLM-Engineers-Handbook Public Notifications You must be signed in to change notification settings Fork 1.2k Star 4.9k Code Issues 23 Pull requests 8 Actions Security and quality 0 Insights Additional navigation options Code Issues Pull requests Actions Security and quality Insights {"payload":{"codeViewRepoRoute":{"path":"/","refInfo":{"name":"main","listCacheKey":"v0:1735803499.0","canEdit":false,"refType":"branch","currentOid":"e6038da300616020695f8effb3cbf0a78aac5870"},"tree":{"items":[{"name":".github/workflows","path":".github/workflows","contentType":"directory","hasSimplifiedPath":true},{"name":".vscode","path":".vscode","contentType":"directory"},{"name":"code_snippets","path":"code_snippets","contentType":"directory"},{"name":"configs","path":"configs","contentType":"directory"},{"name":"data","path":"data","contentType":"directory"},{"name":"images","path":"images","contentType":"directory"},{"name":"llm_engineering","path":"llm_engineering","contentType":"directory"},{"name":"pipelines","path":"pipelines","contentType":"directory"},{"name":"steps","path":"steps","contentType":"directory"},{"name":"tests","path":"tests","contentType":"directory"},{"name":"tools","path":"tools","contentType":"directory"},{"name":".env.example","path":".env.example","contentType":"file"},{"name":".gitignore","path":".gitignore","contentType":"file"},{"name":".pre-commit-config.yaml","path":".pre-commit-config.yaml","contentType":"file"},{"name":".python-version","path":".python-version","contentType":"file"},{"name":"Dockerfile","path":"Dockerfile","contentType":"file"},{"name":"LICENSE","path":"LICENSE","contentType":"file"},{"name":"README.md","path":"README.md","contentType":"file"},{"name":"docker-compose.yml","path":"docker-compose.yml","contentType":"file"},{"name":"poetry.lock","path":"poetry.lock","contentType":"file"},{"name":"pyproject.toml","path":"pyproject.toml","contentType":"file"},{"name":"ruff.toml","path":"ruff.toml","contentType":"file"}],"totalCount":22,"templateDirectorySuggestionUrl":null,"readme":null,"showBranchInfobar":false},"userNameDisplayConfiguration":"handle","treeExpanded":false,"symbolsExpanded":false,"copilotSWEAgentEnabled":false,"isOverview":true,"overview":{"banners":{"shouldRecommendReadme":false,"isPersonalRepo":false,"showUseActionBanner":false,"actionSlug":null,"actionId":null,"showProtectBranchBanner":false,"transactionalMessageBanner":null,"publishBannersInfo":{"dismissActionNoticePath":"/settings/dismiss-notice/publish_action_from_repo","releasePath":"/PacktPublishing/LLM-Engineers-Handbook/releases/new?marketplace=true","showPublishActionBanner":false},"interactionLimitBanner":null,"showInvitationBanner":false,"inviterName":null,"actionsMigrationBannerInfo":{"releaseTags":[],"showImmutableActionsMigrationBanner":false,"initialMigrationStatus":null}},"codeButton":{"contactPath":"/contact","isEnterprise":false,"local":{"protocolInfo":{"httpAvailable":true,"sshAvailable":null,"httpUrl":"https://github.com/PacktPublishing/LLM-Engineers-Handbook.git","showCloneWarning":null,"sshUrl":null,"sshCertificatesRequired":null,"sshCertificatesAvailable":null,"ghCliUrl":"gh repo clone PacktPublishing/LLM-Engineers-Handbook","defaultProtocol":"http","newSshKeyUrl":"/settings/ssh/new","setProtocolPath":"/users/set_protocol"},"platformInfo":{"cloneUrl":"https://desktop.github.com","showVisualStudioCloneButton":false,"visualStudioCloneUrl":"https://windows.github.com","showXcodeCloneButton":false,"xcodeCloneUrl":"xcode://clone?repo=https%3A%2F%2Fgithub.com%2FPacktPublishing%2FLLM-Engineers-Handbook","zipballUrl":"/PacktPublishing/LLM-Engineers-Handbook/archive/refs/heads/main.zip"}},"newCodespacePath":"/codespaces/new?hide_repo_select=true\u0026repo=784183059"},"popovers":{"rename":null,"renamedParentRepo":null},"commitCount":"201","overviewFiles":[{"displayName":"README.md","repoName":"LLM-Engineers-Handbook","refName":"main","path":"README.md","preferredFileType":"readme","tabName":"README","richText":"\u003carticle class=\"markdown-body entry-content container-lg\" itemprop=\"text\"\u003e\u003cp align=\"center\" dir=\"auto\"\u003e\u003ca href=\"https://www.eventbrite.com/e/ship-production-pytorch-system-in-a-day-train-optimize-deploy-workshop-tickets-1983348934052?aff=GitHub\" rel=\"nofollow\"\u003e\u003cimg src=\"https://camo.githubusercontent.com/b9c13fddfcb7d388660e7c60c3f7c67412deef8f3a83833a798b18b4b5efda38/68747470733a2f2f7374617469632e7061636b742d63646e2e636f6d2f6173736574732f696d616765732f7061636b742b6576656e74732f776f726b73686f705f6173686973685f6769746875622e706e67\" data-canonical-src=\"https://static.packt-cdn.com/assets/images/packt+events/workshop_ashish_github.png\" style=\"max-width: 100%;\"\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cp align=\"center\" dir=\"auto\"\u003e\u003ca href=\"https://www.packtpub.com/en-us/unlock?step=1\" rel=\"nofollow\"\u003e\u003cimg src=\"https://camo.githubusercontent.com/1d5684bdb5380c77317e1c2e8805f440f76c0771f523aa0c45a536cdfc10bf1e/68747470733a2f2f7374617469632e7061636b742d63646e2e636f6d2f6173736574732f696d616765732f7061636b742b6576656e74732f66696e616c47485f64657369676e5f72656465656d2e706e67\" data-canonical-src=\"https://static.packt-cdn.com/assets/images/packt+events/finalGH_design_redeem.png\" style=\"max-width: 100%;\"\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cdiv align=\"center\" dir=\"auto\"\u003e\n \u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch1 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003e👷 LLM Engineer's Handbook\u003c/h1\u003e\u003ca id=\"user-content--llm-engineers-handbook\" class=\"anchor\" aria-label=\"Permalink: 👷 LLM Engineer's Handbook\" href=\"#-llm-engineers-handbook\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n \u003cp dir=\"auto\"\u003eOfficial repository of the \u003ca href=\"https://www.amazon.com/LLM-Engineers-Handbook-engineering-production/dp/1836200072/\" rel=\"nofollow\"\u003eLLM Engineer's Handbook\u003c/a\u003e by \u003ca href=\"https://github.com/iusztinpaul\"\u003ePaul Iusztin\u003c/a\u003e and \u003ca href=\"https://github.com/mlabonne\"\u003eMaxime Labonne\u003c/a\u003e\u003c/p\u003e\n\u003c/div\u003e\n\u003cbr\u003e\n\u003cp align=\"center\" dir=\"auto\"\u003e\n \u003ca href=\"https://www.amazon.com/LLM-Engineers-Handbook-engineering-production/dp/1836200072/\" rel=\"nofollow\"\u003e\n \u003cimg src=\"/PacktPublishing/LLM-Engineers-Handbook/raw/main/images/cover_plus.png\" alt=\"Book cover\" style=\"max-width: 100%;\"\u003e\n \u003c/a\u003e\n\u003c/p\u003e\n\u003cp align=\"center\" dir=\"auto\"\u003e\n Find the book on \u003ca href=\"https://www.amazon.com/LLM-Engineers-Handbook-engineering-production/dp/1836200072/\" rel=\"nofollow\"\u003eAmazon\u003c/a\u003e or \u003ca href=\"https://www.packtpub.com/en-us/product/llm-engineers-handbook-9781836200062\" rel=\"nofollow\"\u003ePackt\u003c/a\u003e\n\u003c/p\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch2 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003e🌟 Features\u003c/h2\u003e\u003ca id=\"user-content--features\" class=\"anchor\" aria-label=\"Permalink: 🌟 Features\" href=\"#-features\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eThe goal of this book is to create your own end-to-end LLM-based system using best practices:\u003c/p\u003e\n\u003cul dir=\"auto\"\u003e\n\u003cli\u003e📝 Data collection \u0026amp; generation\u003c/li\u003e\n\u003cli\u003e🔄 LLM training pipeline\u003c/li\u003e\n\u003cli\u003e📊 Simple RAG system\u003c/li\u003e\n\u003cli\u003e🚀 Production-ready AWS deployment\u003c/li\u003e\n\u003cli\u003e🔍 Comprehensive monitoring\u003c/li\u003e\n\u003cli\u003e🧪 Testing and evaluation framework\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp dir=\"auto\"\u003eYou can download and use the final trained model on \u003ca href=\"https://huggingface.co/mlabonne/TwinLlama-3.1-8B-DPO\" rel=\"nofollow\"\u003eHugging Face\u003c/a\u003e.\u003c/p\u003e\n\u003cdiv class=\"markdown-alert markdown-alert-important\" dir=\"auto\"\u003e\u003cp class=\"markdown-alert-title\" dir=\"auto\"\u003e\u003csvg class=\"octicon octicon-report mr-2\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"M0 1.75C0 .784.784 0 1.75 0h12.5C15.216 0 16 .784 16 1.75v9.5A1.75 1.75 0 0 1 14.25 13H8.06l-2.573 2.573A1.458 1.458 0 0 1 3 14.543V13H1.75A1.75 1.75 0 0 1 0 11.25Zm1.75-.25a.25.25 0 0 0-.25.25v9.5c0 .138.112.25.25.25h2a.75.75 0 0 1 .75.75v2.19l2.72-2.72a.749.749 0 0 1 .53-.22h6.5a.25.25 0 0 0 .25-.25v-9.5a.25.25 0 0 0-.25-.25Zm7 2.25v2.5a.75.75 0 0 1-1.5 0v-2.5a.75.75 0 0 1 1.5 0ZM9 9a1 1 0 1 1-2 0 1 1 0 0 1 2 0Z\"\u003e\u003c/path\u003e\u003c/svg\u003eImportant\u003c/p\u003e\u003cp dir=\"auto\"\u003eThe code in this GitHub repository is actively maintained and may contain updates not reflected in the book. \u003cstrong\u003eAlways refer to this repository for the latest version of the code.\u003c/strong\u003e\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch2 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003e🔗 Dependencies\u003c/h2\u003e\u003ca id=\"user-content--dependencies\" class=\"anchor\" aria-label=\"Permalink: 🔗 Dependencies\" href=\"#-dependencies\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch3 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eLocal dependencies\u003c/h3\u003e\u003ca id=\"user-content-local-dependencies\" class=\"anchor\" aria-label=\"Permalink: Local dependencies\" href=\"#local-dependencies\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eTo install and run the project locally, you need the following dependencies.\u003c/p\u003e\n\u003cmarkdown-accessiblity-table\u003e\u003ctable\u003e\n\u003cthead\u003e\n\u003ctr\u003e\n\u003cth\u003eTool\u003c/th\u003e\n\u003cth\u003eVersion\u003c/th\u003e\n\u003cth\u003ePurpose\u003c/th\u003e\n\u003cth\u003eInstallation Link\u003c/th\u003e\n\u003c/tr\u003e\n\u003c/thead\u003e\n\u003ctbody\u003e\n\u003ctr\u003e\n\u003ctd\u003epyenv\u003c/td\u003e\n\u003ctd\u003e≥2.3.36\u003c/td\u003e\n\u003ctd\u003eMultiple Python versions (optional)\u003c/td\u003e\n\u003ctd\u003e\u003ca href=\"https://github.com/pyenv/pyenv?tab=readme-ov-file#installation\"\u003eInstall Guide\u003c/a\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003ePython\u003c/td\u003e\n\u003ctd\u003e3.11\u003c/td\u003e\n\u003ctd\u003eRuntime environment\u003c/td\u003e\n\u003ctd\u003e\u003ca href=\"https://www.python.org/downloads/\" rel=\"nofollow\"\u003eDownload\u003c/a\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003ePoetry\u003c/td\u003e\n\u003ctd\u003e\u0026gt;= 1.8.3 and \u0026lt; 2.0\u003c/td\u003e\n\u003ctd\u003ePackage management\u003c/td\u003e\n\u003ctd\u003e\u003ca href=\"https://python-poetry.org/docs/#installation\" rel=\"nofollow\"\u003eInstall Guide\u003c/a\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003eDocker\u003c/td\u003e\n\u003ctd\u003e≥27.1.1\u003c/td\u003e\n\u003ctd\u003eContainerization\u003c/td\u003e\n\u003ctd\u003e\u003ca href=\"https://docs.docker.com/engine/install/\" rel=\"nofollow\"\u003eInstall Guide\u003c/a\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003eAWS CLI\u003c/td\u003e\n\u003ctd\u003e≥2.15.42\u003c/td\u003e\n\u003ctd\u003eCloud management\u003c/td\u003e\n\u003ctd\u003e\u003ca href=\"https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html\" rel=\"nofollow\"\u003eInstall Guide\u003c/a\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003eGit\u003c/td\u003e\n\u003ctd\u003e≥2.44.0\u003c/td\u003e\n\u003ctd\u003eVersion control\u003c/td\u003e\n\u003ctd\u003e\u003ca href=\"https://git-scm.com/downloads\" rel=\"nofollow\"\u003eDownload\u003c/a\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/tbody\u003e\n\u003c/table\u003e\u003c/markdown-accessiblity-table\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch3 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eCloud services\u003c/h3\u003e\u003ca id=\"user-content-cloud-services\" class=\"anchor\" aria-label=\"Permalink: Cloud services\" href=\"#cloud-services\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eThe code also uses and depends on the following cloud services. For now, you don't have to do anything. We will guide you in the installation and deployment sections on how to use them:\u003c/p\u003e\n\u003cmarkdown-accessiblity-table\u003e\u003ctable\u003e\n\u003cthead\u003e\n\u003ctr\u003e\n\u003cth\u003eService\u003c/th\u003e\n\u003cth\u003ePurpose\u003c/th\u003e\n\u003c/tr\u003e\n\u003c/thead\u003e\n\u003ctbody\u003e\n\u003ctr\u003e\n\u003ctd\u003e\u003ca href=\"https://huggingface.com/\" rel=\"nofollow\"\u003eHuggingFace\u003c/a\u003e\u003c/td\u003e\n\u003ctd\u003eModel registry\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\u003ca href=\"https://www.comet.com/site/products/opik/?utm_source=llm_handbook\u0026amp;utm_medium=github\u0026amp;utm_campaign=opik\" rel=\"nofollow\"\u003eComet ML\u003c/a\u003e\u003c/td\u003e\n\u003ctd\u003eExperiment tracker\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\u003ca href=\"https://www.comet.com/site/products/opik/?utm_source=llm_handbook\u0026amp;utm_medium=github\u0026amp;utm_campaign=opik\" rel=\"nofollow\"\u003eOpik\u003c/a\u003e\u003c/td\u003e\n\u003ctd\u003ePrompt monitoring\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\u003ca href=\"https://www.zenml.io/\" rel=\"nofollow\"\u003eZenML\u003c/a\u003e\u003c/td\u003e\n\u003ctd\u003eOrchestrator and artifacts layer\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\u003ca href=\"https://aws.amazon.com/\" rel=\"nofollow\"\u003eAWS\u003c/a\u003e\u003c/td\u003e\n\u003ctd\u003eCompute and storage\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\u003ca href=\"https://www.mongodb.com/\" rel=\"nofollow\"\u003eMongoDB\u003c/a\u003e\u003c/td\u003e\n\u003ctd\u003eNoSQL database\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\u003ca href=\"https://qdrant.tech/\" rel=\"nofollow\"\u003eQdrant\u003c/a\u003e\u003c/td\u003e\n\u003ctd\u003eVector database\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\u003ca href=\"https://github.com/features/actions\"\u003eGitHub Actions\u003c/a\u003e\u003c/td\u003e\n\u003ctd\u003eCI/CD pipeline\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/tbody\u003e\n\u003c/table\u003e\u003c/markdown-accessiblity-table\u003e\n\u003cp dir=\"auto\"\u003eIn the \u003ca href=\"https://www.amazon.com/LLM-Engineers-Handbook-engineering-production/dp/1836200072/\" rel=\"nofollow\"\u003eLLM Engineer's Handbook\u003c/a\u003e, Chapter 2 will walk you through each tool. Chapters 10 and 11 provide step-by-step guides on how to set up everything you need.\u003c/p\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch2 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003e🗂️ Project Structure\u003c/h2\u003e\u003ca id=\"user-content-️-project-structure\" class=\"anchor\" aria-label=\"Permalink: 🗂️ Project Structure\" href=\"#️-project-structure\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eHere is the directory overview:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\".\n├── code_snippets/ # Standalone example code\n├── configs/ # Pipeline configuration files\n├── llm_engineering/ # Core project package\n│ ├── application/ \n│ ├── domain/ \n│ ├── infrastructure/ \n│ ├── model/ \n├── pipelines/ # ML pipeline definitions\n├── steps/ # Pipeline components\n├── tests/ # Test examples\n├── tools/ # Utility scripts\n│ ├── run.py\n│ ├── ml_service.py\n│ ├── rag.py\n│ ├── data_warehouse.py\"\u003e\u003cpre\u003e\u003cspan class=\"pl-c1\"\u003e.\u003c/span\u003e\n├── code_snippets/ \u003cspan class=\"pl-c\"\u003e\u003cspan class=\"pl-c\"\u003e#\u003c/span\u003e Standalone example code\u003c/span\u003e\n├── configs/ \u003cspan class=\"pl-c\"\u003e\u003cspan class=\"pl-c\"\u003e#\u003c/span\u003e Pipeline configuration files\u003c/span\u003e\n├── llm_engineering/ \u003cspan class=\"pl-c\"\u003e\u003cspan class=\"pl-c\"\u003e#\u003c/span\u003e Core project package\u003c/span\u003e\n│ ├── application/ \n│ ├── domain/ \n│ ├── infrastructure/ \n│ ├── model/ \n├── pipelines/ \u003cspan class=\"pl-c\"\u003e\u003cspan class=\"pl-c\"\u003e#\u003c/span\u003e ML pipeline definitions\u003c/span\u003e\n├── steps/ \u003cspan class=\"pl-c\"\u003e\u003cspan class=\"pl-c\"\u003e#\u003c/span\u003e Pipeline components\u003c/span\u003e\n├── tests/ \u003cspan class=\"pl-c\"\u003e\u003cspan class=\"pl-c\"\u003e#\u003c/span\u003e Test examples\u003c/span\u003e\n├── tools/ \u003cspan class=\"pl-c\"\u003e\u003cspan class=\"pl-c\"\u003e#\u003c/span\u003e Utility scripts\u003c/span\u003e\n│ ├── run.py\n│ ├── ml_service.py\n│ ├── rag.py\n│ ├── data_warehouse.py\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003e\u003ccode\u003ellm_engineering/\u003c/code\u003e is the main Python package implementing LLM and RAG functionality. It follows Domain-Driven Design (DDD) principles:\u003c/p\u003e\n\u003cul dir=\"auto\"\u003e\n\u003cli\u003e\u003ccode\u003edomain/\u003c/code\u003e: Core business entities and structures\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eapplication/\u003c/code\u003e: Business logic, crawlers, and RAG implementation\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003emodel/\u003c/code\u003e: LLM training and inference\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003einfrastructure/\u003c/code\u003e: External service integrations (AWS, Qdrant, MongoDB, FastAPI)\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp dir=\"auto\"\u003eThe code logic and imports flow as follows: \u003ccode\u003einfrastructure\u003c/code\u003e → \u003ccode\u003emodel\u003c/code\u003e → \u003ccode\u003eapplication\u003c/code\u003e → \u003ccode\u003edomain\u003c/code\u003e\u003c/p\u003e\n\u003cp dir=\"auto\"\u003e\u003ccode\u003epipelines/\u003c/code\u003e: Contains the ZenML ML pipelines, which serve as the entry point for all the ML pipelines. Coordinates the data processing and model training stages of the ML lifecycle.\u003c/p\u003e\n\u003cp dir=\"auto\"\u003e\u003ccode\u003esteps/\u003c/code\u003e: Contains individual ZenML steps, which are reusable components for building and customizing ZenML pipelines. Steps perform specific tasks (e.g., data loading, preprocessing) and can be combined within the ML pipelines.\u003c/p\u003e\n\u003cp dir=\"auto\"\u003e\u003ccode\u003etests/\u003c/code\u003e: Covers a few sample tests used as examples within the CI pipeline.\u003c/p\u003e\n\u003cp dir=\"auto\"\u003e\u003ccode\u003etools/\u003c/code\u003e: Utility scripts used to call the ZenML pipelines and inference code:\u003c/p\u003e\n\u003cul dir=\"auto\"\u003e\n\u003cli\u003e\u003ccode\u003erun.py\u003c/code\u003e: Entry point script to run ZenML pipelines.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eml_service.py\u003c/code\u003e: Starts the REST API inference server.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003erag.py\u003c/code\u003e: Demonstrates usage of the RAG retrieval module.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003edata_warehouse.py\u003c/code\u003e: Used to export or import data from the MongoDB data warehouse through JSON files.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp dir=\"auto\"\u003e\u003ccode\u003econfigs/\u003c/code\u003e: ZenML YAML configuration files to control the execution of pipelines and steps.\u003c/p\u003e\n\u003cp dir=\"auto\"\u003e\u003ccode\u003ecode_snippets/\u003c/code\u003e: Independent code examples that can be executed independently.\u003c/p\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch2 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003e💻 Installation\u003c/h2\u003e\u003ca id=\"user-content--installation\" class=\"anchor\" aria-label=\"Permalink: 💻 Installation\" href=\"#-installation\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cdiv class=\"markdown-alert markdown-alert-note\" dir=\"auto\"\u003e\u003cp class=\"markdown-alert-title\" dir=\"auto\"\u003e\u003csvg class=\"octicon octicon-info mr-2\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"M0 8a8 8 0 1 1 16 0A8 8 0 0 1 0 8Zm8-6.5a6.5 6.5 0 1 0 0 13 6.5 6.5 0 0 0 0-13ZM6.5 7.75A.75.75 0 0 1 7.25 7h1a.75.75 0 0 1 .75.75v2.75h.25a.75.75 0 0 1 0 1.5h-2a.75.75 0 0 1 0-1.5h.25v-2h-.25a.75.75 0 0 1-.75-.75ZM8 6a1 1 0 1 1 0-2 1 1 0 0 1 0 2Z\"\u003e\u003c/path\u003e\u003c/svg\u003eNote\u003c/p\u003e\u003cp dir=\"auto\"\u003eIf you are experiencing issues while installing and running the repository, consider checking the \u003ca href=\"https://github.com/PacktPublishing/LLM-Engineers-Handbook/issues\"\u003eIssues\u003c/a\u003e GitHub section for other people who solved similar problems or directly asking us for help.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch3 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003e1. Clone the Repository\u003c/h3\u003e\u003ca id=\"user-content-1-clone-the-repository\" class=\"anchor\" aria-label=\"Permalink: 1. Clone the Repository\" href=\"#1-clone-the-repository\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eStart by cloning the repository and navigating to the project directory:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"git clone https://github.com/PacktPublishing/LLM-Engineers-Handbook.git\ncd LLM-Engineers-Handbook \"\u003e\u003cpre\u003egit clone https://github.com/PacktPublishing/LLM-Engineers-Handbook.git\n\u003cspan class=\"pl-c1\"\u003ecd\u003c/span\u003e LLM-Engineers-Handbook \u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eNext, we have to prepare your Python environment and its adjacent dependencies.\u003c/p\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch3 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003e2. Set Up Python Environment\u003c/h3\u003e\u003ca id=\"user-content-2-set-up-python-environment\" class=\"anchor\" aria-label=\"Permalink: 2. Set Up Python Environment\" href=\"#2-set-up-python-environment\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eThe project requires Python 3.11. You can either use your global Python installation or set up a project-specific version using pyenv.\u003c/p\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch4 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eOption A: Using Global Python (if version 3.11 is installed)\u003c/h4\u003e\u003ca id=\"user-content-option-a-using-global-python-if-version-311-is-installed\" class=\"anchor\" aria-label=\"Permalink: Option A: Using Global Python (if version 3.11 is installed)\" href=\"#option-a-using-global-python-if-version-311-is-installed\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eVerify your Python version:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"python --version # Should show Python 3.11.x\"\u003e\u003cpre\u003epython --version \u003cspan class=\"pl-c\"\u003e\u003cspan class=\"pl-c\"\u003e#\u003c/span\u003e Should show Python 3.11.x\u003c/span\u003e\u003c/pre\u003e\u003c/div\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch4 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eOption B: Using pyenv (recommended)\u003c/h4\u003e\u003ca id=\"user-content-option-b-using-pyenv-recommended\" class=\"anchor\" aria-label=\"Permalink: Option B: Using pyenv (recommended)\" href=\"#option-b-using-pyenv-recommended\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003col dir=\"auto\"\u003e\n\u003cli\u003eVerify pyenv installation:\u003c/li\u003e\n\u003c/ol\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"pyenv --version # Should show pyenv 2.3.36 or later\"\u003e\u003cpre\u003epyenv --version \u003cspan class=\"pl-c\"\u003e\u003cspan class=\"pl-c\"\u003e#\u003c/span\u003e Should show pyenv 2.3.36 or later\u003c/span\u003e\u003c/pre\u003e\u003c/div\u003e\n\u003col start=\"2\" dir=\"auto\"\u003e\n\u003cli\u003eInstall Python 3.11.8:\u003c/li\u003e\n\u003c/ol\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"pyenv install 3.11.8\"\u003e\u003cpre\u003epyenv install 3.11.8\u003c/pre\u003e\u003c/div\u003e\n\u003col start=\"3\" dir=\"auto\"\u003e\n\u003cli\u003eVerify the installation:\u003c/li\u003e\n\u003c/ol\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"python --version # Should show Python 3.11.8\"\u003e\u003cpre\u003epython --version \u003cspan class=\"pl-c\"\u003e\u003cspan class=\"pl-c\"\u003e#\u003c/span\u003e Should show Python 3.11.8\u003c/span\u003e\u003c/pre\u003e\u003c/div\u003e\n\u003col start=\"4\" dir=\"auto\"\u003e\n\u003cli\u003eConfirm Python version in the project directory:\u003c/li\u003e\n\u003c/ol\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"python --version\n# Output: Python 3.11.8\"\u003e\u003cpre\u003epython --version\n\u003cspan class=\"pl-c\"\u003e\u003cspan class=\"pl-c\"\u003e#\u003c/span\u003e Output: Python 3.11.8\u003c/span\u003e\u003c/pre\u003e\u003c/div\u003e\n\u003cdiv class=\"markdown-alert markdown-alert-note\" dir=\"auto\"\u003e\u003cp class=\"markdown-alert-title\" dir=\"auto\"\u003e\u003csvg class=\"octicon octicon-info mr-2\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"M0 8a8 8 0 1 1 16 0A8 8 0 0 1 0 8Zm8-6.5a6.5 6.5 0 1 0 0 13 6.5 6.5 0 0 0 0-13ZM6.5 7.75A.75.75 0 0 1 7.25 7h1a.75.75 0 0 1 .75.75v2.75h.25a.75.75 0 0 1 0 1.5h-2a.75.75 0 0 1 0-1.5h.25v-2h-.25a.75.75 0 0 1-.75-.75ZM8 6a1 1 0 1 1 0-2 1 1 0 0 1 0 2Z\"\u003e\u003c/path\u003e\u003c/svg\u003eNote\u003c/p\u003e\u003cp dir=\"auto\"\u003eThe project includes a \u003ccode\u003e.python-version\u003c/code\u003e file that automatically sets the correct Python version when you're in the project directory.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch3 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003e3. Install Dependencies\u003c/h3\u003e\u003ca id=\"user-content-3-install-dependencies\" class=\"anchor\" aria-label=\"Permalink: 3. Install Dependencies\" href=\"#3-install-dependencies\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eThe project uses Poetry for dependency management.\u003c/p\u003e\n\u003col dir=\"auto\"\u003e\n\u003cli\u003eVerify Poetry installation:\u003c/li\u003e\n\u003c/ol\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry --version # Should show Poetry version 1.8.3 or later\"\u003e\u003cpre\u003epoetry --version \u003cspan class=\"pl-c\"\u003e\u003cspan class=\"pl-c\"\u003e#\u003c/span\u003e Should show Poetry version 1.8.3 or later\u003c/span\u003e\u003c/pre\u003e\u003c/div\u003e\n\u003col start=\"2\" dir=\"auto\"\u003e\n\u003cli\u003eSet up the project environment and install dependencies:\u003c/li\u003e\n\u003c/ol\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry env use 3.11\npoetry install --without aws\npoetry run pre-commit install\"\u003e\u003cpre\u003epoetry env use 3.11\npoetry install --without aws\npoetry run pre-commit install\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eThis will:\u003c/p\u003e\n\u003cul dir=\"auto\"\u003e\n\u003cli\u003eConfigure Poetry to use Python 3.11\u003c/li\u003e\n\u003cli\u003eInstall project dependencies (excluding AWS-specific packages)\u003c/li\u003e\n\u003cli\u003eSet up pre-commit hooks for code verification\u003c/li\u003e\n\u003c/ul\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch3 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003e4. Activate the Environment\u003c/h3\u003e\u003ca id=\"user-content-4-activate-the-environment\" class=\"anchor\" aria-label=\"Permalink: 4. Activate the Environment\" href=\"#4-activate-the-environment\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eAs our task manager, we run all the scripts using \u003ca href=\"https://poethepoet.natn.io/index.html\" rel=\"nofollow\"\u003ePoe the Poet\u003c/a\u003e.\u003c/p\u003e\n\u003col dir=\"auto\"\u003e\n\u003cli\u003eStart a Poetry shell:\u003c/li\u003e\n\u003c/ol\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry shell\"\u003e\u003cpre\u003epoetry shell\u003c/pre\u003e\u003c/div\u003e\n\u003col start=\"2\" dir=\"auto\"\u003e\n\u003cli\u003eRun project commands using Poe the Poet:\u003c/li\u003e\n\u003c/ol\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe ...\"\u003e\u003cpre\u003epoetry poe ...\u003c/pre\u003e\u003c/div\u003e\n\u003cdetails\u003e\n\u003csummary\u003e🔧 Troubleshooting Poe the Poet Installation\u003c/summary\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch3 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eAlternative Command Execution\u003c/h3\u003e\u003ca id=\"user-content-alternative-command-execution\" class=\"anchor\" aria-label=\"Permalink: Alternative Command Execution\" href=\"#alternative-command-execution\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eIf you're experiencing issues with \u003ccode\u003epoethepoet\u003c/code\u003e, you can still run the project commands directly through Poetry. Here's how:\u003c/p\u003e\n\u003col dir=\"auto\"\u003e\n\u003cli\u003eLook up the command definition in \u003ccode\u003epyproject.toml\u003c/code\u003e\u003c/li\u003e\n\u003cli\u003eUse \u003ccode\u003epoetry run\u003c/code\u003e with the underlying command\u003c/li\u003e\n\u003c/ol\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch4 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eExample:\u003c/h4\u003e\u003ca id=\"user-content-example\" class=\"anchor\" aria-label=\"Permalink: Example:\" href=\"#example\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eInstead of:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe local-infrastructure-up\"\u003e\u003cpre\u003epoetry poe local-infrastructure-up\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eUse the direct command from pyproject.toml:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry run \u0026lt;actual-command-from-pyproject-toml\u0026gt;\"\u003e\u003cpre\u003epoetry run \u003cspan class=\"pl-k\"\u003e\u0026lt;\u003c/span\u003eactual-command-from-pyproject-toml\u003cspan class=\"pl-k\"\u003e\u0026gt;\u003c/span\u003e\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eNote: All project commands are defined in the [tool.poe.tasks] section of pyproject.toml\u003c/p\u003e\n\u003c/details\u003e\n\u003cp dir=\"auto\"\u003eNow, let's configure our local project with all the necessary credentials and tokens to run the code locally.\u003c/p\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch3 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003e5. Local Development Setup\u003c/h3\u003e\u003ca id=\"user-content-5-local-development-setup\" class=\"anchor\" aria-label=\"Permalink: 5. Local Development Setup\" href=\"#5-local-development-setup\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eAfter you have installed all the dependencies, you must create and fill a \u003ccode\u003e.env\u003c/code\u003e file with your credentials to appropriately interact with other services and run the project. Setting your sensitive credentials in a \u003ccode\u003e.env\u003c/code\u003e file is a good security practice, as this file won't be committed to GitHub or shared with anyone else.\u003c/p\u003e\n\u003col dir=\"auto\"\u003e\n\u003cli\u003eFirst, copy our example by running the following:\u003c/li\u003e\n\u003c/ol\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"cp .env.example .env # The file must be at your repository's root!\"\u003e\u003cpre\u003ecp .env.example .env \u003cspan class=\"pl-c\"\u003e\u003cspan class=\"pl-c\"\u003e#\u003c/span\u003e The file must be at your repository's root!\u003c/span\u003e\u003c/pre\u003e\u003c/div\u003e\n\u003col start=\"2\" dir=\"auto\"\u003e\n\u003cli\u003eNow, let's understand how to fill in all the essential variables within the \u003ccode\u003e.env\u003c/code\u003e file to get you started. The following are the mandatory settings we must complete when working locally:\u003c/li\u003e\n\u003c/ol\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch4 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eOpenAI\u003c/h4\u003e\u003ca id=\"user-content-openai\" class=\"anchor\" aria-label=\"Permalink: OpenAI\" href=\"#openai\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eTo authenticate to OpenAI's API, you must fill out the \u003ccode\u003eOPENAI_API_KEY\u003c/code\u003e env var with an authentication token.\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-dotenv notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"OPENAI_API_KEY=your_api_key_here\"\u003e\u003cpre\u003e\u003cspan class=\"pl-v\"\u003eOPENAI_API_KEY\u003c/span\u003e\u003cspan class=\"pl-k\"\u003e=\u003c/span\u003e\u003cspan class=\"pl-s\"\u003eyour_api_key_here\u003c/span\u003e\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003e→ Check out this \u003ca href=\"https://platform.openai.com/docs/quickstart\" rel=\"nofollow\"\u003etutorial\u003c/a\u003e to learn how to provide one from OpenAI.\u003c/p\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch4 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eHugging Face\u003c/h4\u003e\u003ca id=\"user-content-hugging-face\" class=\"anchor\" aria-label=\"Permalink: Hugging Face\" href=\"#hugging-face\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eTo authenticate to Hugging Face, you must fill out the \u003ccode\u003eHUGGINGFACE_ACCESS_TOKEN\u003c/code\u003e env var with an authentication token.\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-dotenv notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"HUGGINGFACE_ACCESS_TOKEN=your_token_here\"\u003e\u003cpre\u003e\u003cspan class=\"pl-v\"\u003eHUGGINGFACE_ACCESS_TOKEN\u003c/span\u003e\u003cspan class=\"pl-k\"\u003e=\u003c/span\u003e\u003cspan class=\"pl-s\"\u003eyour_token_here\u003c/span\u003e\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003e→ Check out this \u003ca href=\"https://huggingface.co/docs/hub/en/security-tokens\" rel=\"nofollow\"\u003etutorial\u003c/a\u003e to learn how to provide one from Hugging Face.\u003c/p\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch4 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eComet ML \u0026amp; Opik\u003c/h4\u003e\u003ca id=\"user-content-comet-ml--opik\" class=\"anchor\" aria-label=\"Permalink: Comet ML \u0026amp; Opik\" href=\"#comet-ml--opik\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eTo authenticate to Comet ML (required only during training) and Opik, you must fill out the \u003ccode\u003eCOMET_API_KEY\u003c/code\u003e env var with your authentication token.\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-dotenv notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"COMET_API_KEY=your_api_key_here\"\u003e\u003cpre\u003e\u003cspan class=\"pl-v\"\u003eCOMET_API_KEY\u003c/span\u003e\u003cspan class=\"pl-k\"\u003e=\u003c/span\u003e\u003cspan class=\"pl-s\"\u003eyour_api_key_here\u003c/span\u003e\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003e→ Check out this \u003ca href=\"https://www.comet.com/docs/opik/?utm_source=llm_handbook\u0026amp;utm_medium=github\u0026amp;utm_campaign=opik\" rel=\"nofollow\"\u003etutorial\u003c/a\u003e to learn how to get started with Opik. You can also access Opik's dashboard using 🔗\u003ca href=\"https://www.comet.com/opik?utm_source=llm_handbook\u0026amp;utm_medium=github\u0026amp;utm_content=opik\" rel=\"nofollow\"\u003ethis link\u003c/a\u003e.\u003c/p\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch3 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003e6. Deployment Setup\u003c/h3\u003e\u003ca id=\"user-content-6-deployment-setup\" class=\"anchor\" aria-label=\"Permalink: 6. Deployment Setup\" href=\"#6-deployment-setup\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eWhen deploying the project to the cloud, we must set additional settings for Mongo, Qdrant, and AWS. If you are just working locally, the default values of these env vars will work out of the box. Detailed deployment instructions are available in Chapter 11 of the \u003ca href=\"https://www.amazon.com/LLM-Engineers-Handbook-engineering-production/dp/1836200072/\" rel=\"nofollow\"\u003eLLM Engineer's Handbook\u003c/a\u003e.\u003c/p\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch4 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eMongoDB\u003c/h4\u003e\u003ca id=\"user-content-mongodb\" class=\"anchor\" aria-label=\"Permalink: MongoDB\" href=\"#mongodb\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eWe must change the \u003ccode\u003eDATABASE_HOST\u003c/code\u003e env var with the URL pointing to your cloud MongoDB cluster.\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-dotenv notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"DATABASE_HOST=your_mongodb_url\"\u003e\u003cpre\u003e\u003cspan class=\"pl-v\"\u003eDATABASE_HOST\u003c/span\u003e\u003cspan class=\"pl-k\"\u003e=\u003c/span\u003e\u003cspan class=\"pl-s\"\u003eyour_mongodb_url\u003c/span\u003e\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003e→ Check out this \u003ca href=\"https://www.mongodb.com/resources/products/fundamentals/mongodb-cluster-setup\" rel=\"nofollow\"\u003etutorial\u003c/a\u003e to learn how to create and host a MongoDB cluster for free.\u003c/p\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch4 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eQdrant\u003c/h4\u003e\u003ca id=\"user-content-qdrant\" class=\"anchor\" aria-label=\"Permalink: Qdrant\" href=\"#qdrant\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eChange \u003ccode\u003eUSE_QDRANT_CLOUD\u003c/code\u003e to \u003ccode\u003etrue\u003c/code\u003e, \u003ccode\u003eQDRANT_CLOUD_URL\u003c/code\u003e with the URL point to your cloud Qdrant cluster, and \u003ccode\u003eQDRANT_APIKEY\u003c/code\u003e with its API key.\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-dotenv notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"USE_QDRANT_CLOUD=true\nQDRANT_CLOUD_URL=your_qdrant_cloud_url\nQDRANT_APIKEY=your_qdrant_api_key\"\u003e\u003cpre\u003e\u003cspan class=\"pl-v\"\u003eUSE_QDRANT_CLOUD\u003c/span\u003e\u003cspan class=\"pl-k\"\u003e=\u003c/span\u003e\u003cspan class=\"pl-s\"\u003etrue\u003c/span\u003e\n\u003cspan class=\"pl-v\"\u003eQDRANT_CLOUD_URL\u003c/span\u003e\u003cspan class=\"pl-k\"\u003e=\u003c/span\u003e\u003cspan class=\"pl-s\"\u003eyour_qdrant_cloud_url\u003c/span\u003e\n\u003cspan class=\"pl-v\"\u003eQDRANT_APIKEY\u003c/span\u003e\u003cspan class=\"pl-k\"\u003e=\u003c/span\u003e\u003cspan class=\"pl-s\"\u003eyour_qdrant_api_key\u003c/span\u003e\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003e→ Check out this \u003ca href=\"https://qdrant.tech/documentation/cloud/create-cluster/\" rel=\"nofollow\"\u003etutorial\u003c/a\u003e to learn how to create a Qdrant cluster for free\u003c/p\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch4 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eAWS\u003c/h4\u003e\u003ca id=\"user-content-aws\" class=\"anchor\" aria-label=\"Permalink: AWS\" href=\"#aws\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eFor your AWS set-up to work correctly, you need the AWS CLI installed on your local machine and properly configured with an admin user (or a user with enough permissions to create new SageMaker, ECR, and S3 resources; using an admin user will make everything more straightforward).\u003c/p\u003e\n\u003cp dir=\"auto\"\u003eChapter 2 provides step-by-step instructions on how to install the AWS CLI, create an admin user on AWS, and get an access key to set up the \u003ccode\u003eAWS_ACCESS_KEY\u003c/code\u003e and \u003ccode\u003eAWS_SECRET_KEY\u003c/code\u003e environment variables. If you already have an AWS admin user in place, you have to configure the following env vars in your \u003ccode\u003e.env\u003c/code\u003e file:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"AWS_REGION=eu-central-1 # Change it with your AWS region.\nAWS_ACCESS_KEY=your_aws_access_key\nAWS_SECRET_KEY=your_aws_secret_key\"\u003e\u003cpre\u003eAWS_REGION=eu-central-1 \u003cspan class=\"pl-c\"\u003e\u003cspan class=\"pl-c\"\u003e#\u003c/span\u003e Change it with your AWS region.\u003c/span\u003e\nAWS_ACCESS_KEY=your_aws_access_key\nAWS_SECRET_KEY=your_aws_secret_key\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eAWS credentials are typically stored in \u003ccode\u003e~/.aws/credentials\u003c/code\u003e. You can view this file directly using \u003ccode\u003ecat\u003c/code\u003e or similar commands:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"cat ~/.aws/credentials\"\u003e\u003cpre\u003ecat \u003cspan class=\"pl-k\"\u003e~\u003c/span\u003e/.aws/credentials\u003c/pre\u003e\u003c/div\u003e\n\u003cdiv class=\"markdown-alert markdown-alert-important\" dir=\"auto\"\u003e\u003cp class=\"markdown-alert-title\" dir=\"auto\"\u003e\u003csvg class=\"octicon octicon-report mr-2\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"M0 1.75C0 .784.784 0 1.75 0h12.5C15.216 0 16 .784 16 1.75v9.5A1.75 1.75 0 0 1 14.25 13H8.06l-2.573 2.573A1.458 1.458 0 0 1 3 14.543V13H1.75A1.75 1.75 0 0 1 0 11.25Zm1.75-.25a.25.25 0 0 0-.25.25v9.5c0 .138.112.25.25.25h2a.75.75 0 0 1 .75.75v2.19l2.72-2.72a.749.749 0 0 1 .53-.22h6.5a.25.25 0 0 0 .25-.25v-9.5a.25.25 0 0 0-.25-.25Zm7 2.25v2.5a.75.75 0 0 1-1.5 0v-2.5a.75.75 0 0 1 1.5 0ZM9 9a1 1 0 1 1-2 0 1 1 0 0 1 2 0Z\"\u003e\u003c/path\u003e\u003c/svg\u003eImportant\u003c/p\u003e\u003cp dir=\"auto\"\u003eAdditional configuration options are available in \u003ca href=\"https://github.com/PacktPublishing/LLM-Engineers-Handbook/blob/main/llm_engineering/settings.py\"\u003esettings.py\u003c/a\u003e. Any variable in the \u003ccode\u003eSettings\u003c/code\u003e class can be configured through the \u003ccode\u003e.env\u003c/code\u003e file.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch2 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003e🏗️ Infrastructure\u003c/h2\u003e\u003ca id=\"user-content-️-infrastructure\" class=\"anchor\" aria-label=\"Permalink: 🏗️ Infrastructure\" href=\"#️-infrastructure\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch3 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eLocal infrastructure (for testing and development)\u003c/h3\u003e\u003ca id=\"user-content-local-infrastructure-for-testing-and-development\" class=\"anchor\" aria-label=\"Permalink: Local infrastructure (for testing and development)\" href=\"#local-infrastructure-for-testing-and-development\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eWhen running the project locally, we host a MongoDB and Qdrant database using Docker. Also, a testing ZenML server is made available through their Python package.\u003c/p\u003e\n\u003cdiv class=\"markdown-alert markdown-alert-warning\" dir=\"auto\"\u003e\u003cp class=\"markdown-alert-title\" dir=\"auto\"\u003e\u003csvg class=\"octicon octicon-alert mr-2\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"M6.457 1.047c.659-1.234 2.427-1.234 3.086 0l6.082 11.378A1.75 1.75 0 0 1 14.082 15H1.918a1.75 1.75 0 0 1-1.543-2.575Zm1.763.707a.25.25 0 0 0-.44 0L1.698 13.132a.25.25 0 0 0 .22.368h12.164a.25.25 0 0 0 .22-.368Zm.53 3.996v2.5a.75.75 0 0 1-1.5 0v-2.5a.75.75 0 0 1 1.5 0ZM9 11a1 1 0 1 1-2 0 1 1 0 0 1 2 0Z\"\u003e\u003c/path\u003e\u003c/svg\u003eWarning\u003c/p\u003e\u003cp dir=\"auto\"\u003eYou need Docker installed (\u0026gt;= v27.1.1)\u003c/p\u003e\n\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eFor ease of use, you can start the whole local development infrastructure with the following command:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe local-infrastructure-up\"\u003e\u003cpre\u003epoetry poe local-infrastructure-up\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eAlso, you can stop the ZenML server and all the Docker containers using the following command:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe local-infrastructure-down\"\u003e\u003cpre\u003epoetry poe local-infrastructure-down\u003c/pre\u003e\u003c/div\u003e\n\u003cdiv class=\"markdown-alert markdown-alert-warning\" dir=\"auto\"\u003e\u003cp class=\"markdown-alert-title\" dir=\"auto\"\u003e\u003csvg class=\"octicon octicon-alert mr-2\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"M6.457 1.047c.659-1.234 2.427-1.234 3.086 0l6.082 11.378A1.75 1.75 0 0 1 14.082 15H1.918a1.75 1.75 0 0 1-1.543-2.575Zm1.763.707a.25.25 0 0 0-.44 0L1.698 13.132a.25.25 0 0 0 .22.368h12.164a.25.25 0 0 0 .22-.368Zm.53 3.996v2.5a.75.75 0 0 1-1.5 0v-2.5a.75.75 0 0 1 1.5 0ZM9 11a1 1 0 1 1-2 0 1 1 0 0 1 2 0Z\"\u003e\u003c/path\u003e\u003c/svg\u003eWarning\u003c/p\u003e\u003cp dir=\"auto\"\u003eWhen running on MacOS, before starting the server, export the following environment variable:\n\u003ccode\u003eexport OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES\u003c/code\u003e\nOtherwise, the connection between the local server and pipeline will break. 🔗 More details in \u003ca href=\"https://github.com/zenml-io/zenml/issues/2369\" data-hovercard-type=\"issue\" data-hovercard-url=\"/zenml-io/zenml/issues/2369/hovercard\"\u003ethis issue\u003c/a\u003e.\nThis is done by default when using Poe the Poet.\u003c/p\u003e\n\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eStart the inference real-time RESTful API:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe run-inference-ml-service\"\u003e\u003cpre\u003epoetry poe run-inference-ml-service\u003c/pre\u003e\u003c/div\u003e\n\u003cdiv class=\"markdown-alert markdown-alert-important\" dir=\"auto\"\u003e\u003cp class=\"markdown-alert-title\" dir=\"auto\"\u003e\u003csvg class=\"octicon octicon-report mr-2\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"M0 1.75C0 .784.784 0 1.75 0h12.5C15.216 0 16 .784 16 1.75v9.5A1.75 1.75 0 0 1 14.25 13H8.06l-2.573 2.573A1.458 1.458 0 0 1 3 14.543V13H1.75A1.75 1.75 0 0 1 0 11.25Zm1.75-.25a.25.25 0 0 0-.25.25v9.5c0 .138.112.25.25.25h2a.75.75 0 0 1 .75.75v2.19l2.72-2.72a.749.749 0 0 1 .53-.22h6.5a.25.25 0 0 0 .25-.25v-9.5a.25.25 0 0 0-.25-.25Zm7 2.25v2.5a.75.75 0 0 1-1.5 0v-2.5a.75.75 0 0 1 1.5 0ZM9 9a1 1 0 1 1-2 0 1 1 0 0 1 2 0Z\"\u003e\u003c/path\u003e\u003c/svg\u003eImportant\u003c/p\u003e\u003cp dir=\"auto\"\u003eThe LLM microservice, called by the RESTful API, will work only after deploying the LLM to AWS SageMaker.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch4 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eZenML\u003c/h4\u003e\u003ca id=\"user-content-zenml\" class=\"anchor\" aria-label=\"Permalink: ZenML\" href=\"#zenml\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eDashboard URL: \u003ccode\u003elocalhost:8237\u003c/code\u003e\u003c/p\u003e\n\u003cp dir=\"auto\"\u003eDefault credentials:\u003c/p\u003e\n\u003cul dir=\"auto\"\u003e\n\u003cli\u003e\u003ccode\u003eusername\u003c/code\u003e: default\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003epassword\u003c/code\u003e:\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp dir=\"auto\"\u003e→ Find out more about using and setting up \u003ca href=\"https://docs.zenml.io/\" rel=\"nofollow\"\u003eZenML\u003c/a\u003e.\u003c/p\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch4 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eQdrant\u003c/h4\u003e\u003ca id=\"user-content-qdrant-1\" class=\"anchor\" aria-label=\"Permalink: Qdrant\" href=\"#qdrant-1\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eREST API URL: \u003ccode\u003elocalhost:6333\u003c/code\u003e\u003c/p\u003e\n\u003cp dir=\"auto\"\u003eDashboard URL: \u003ccode\u003elocalhost:6333/dashboard\u003c/code\u003e\u003c/p\u003e\n\u003cp dir=\"auto\"\u003e→ Find out more about using and setting up \u003ca href=\"https://qdrant.tech/documentation/quick-start/\" rel=\"nofollow\"\u003eQdrant with Docker\u003c/a\u003e.\u003c/p\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch4 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eMongoDB\u003c/h4\u003e\u003ca id=\"user-content-mongodb-1\" class=\"anchor\" aria-label=\"Permalink: MongoDB\" href=\"#mongodb-1\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eDatabase URI: \u003ccode\u003emongodb://llm_engineering:[email protected]:27017\u003c/code\u003e\u003c/p\u003e\n\u003cp dir=\"auto\"\u003eDatabase name: \u003ccode\u003etwin\u003c/code\u003e\u003c/p\u003e\n\u003cp dir=\"auto\"\u003eDefault credentials:\u003c/p\u003e\n\u003cul dir=\"auto\"\u003e\n\u003cli\u003e\u003ccode\u003eusername\u003c/code\u003e: llm_engineering\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003epassword\u003c/code\u003e: llm_engineering\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp dir=\"auto\"\u003e→ Find out more about using and setting up \u003ca href=\"https://www.mongodb.com/docs/manual/tutorial/install-mongodb-community-with-docker\" rel=\"nofollow\"\u003eMongoDB with Docker\u003c/a\u003e.\u003c/p\u003e\n\u003cp dir=\"auto\"\u003eYou can search your MongoDB collections using your \u003cstrong\u003eIDEs MongoDB plugin\u003c/strong\u003e (which you have to install separately), where you have to use the database URI to connect to the MongoDB database hosted within the Docker container: \u003ccode\u003emongodb://llm_engineering:[email protected]:27017\u003c/code\u003e\u003c/p\u003e\n\u003cdiv class=\"markdown-alert markdown-alert-important\" dir=\"auto\"\u003e\u003cp class=\"markdown-alert-title\" dir=\"auto\"\u003e\u003csvg class=\"octicon octicon-report mr-2\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"M0 1.75C0 .784.784 0 1.75 0h12.5C15.216 0 16 .784 16 1.75v9.5A1.75 1.75 0 0 1 14.25 13H8.06l-2.573 2.573A1.458 1.458 0 0 1 3 14.543V13H1.75A1.75 1.75 0 0 1 0 11.25Zm1.75-.25a.25.25 0 0 0-.25.25v9.5c0 .138.112.25.25.25h2a.75.75 0 0 1 .75.75v2.19l2.72-2.72a.749.749 0 0 1 .53-.22h6.5a.25.25 0 0 0 .25-.25v-9.5a.25.25 0 0 0-.25-.25Zm7 2.25v2.5a.75.75 0 0 1-1.5 0v-2.5a.75.75 0 0 1 1.5 0ZM9 9a1 1 0 1 1-2 0 1 1 0 0 1 2 0Z\"\u003e\u003c/path\u003e\u003c/svg\u003eImportant\u003c/p\u003e\u003cp dir=\"auto\"\u003eEverything related to training or running the LLMs (e.g., training, evaluation, inference) can only be run if you set up AWS SageMaker, as explained in the next section on cloud infrastructure.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch3 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eCloud infrastructure (for production)\u003c/h3\u003e\u003ca id=\"user-content-cloud-infrastructure-for-production\" class=\"anchor\" aria-label=\"Permalink: Cloud infrastructure (for production)\" href=\"#cloud-infrastructure-for-production\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eHere we will quickly present how to deploy the project to AWS and other serverless services. We won't go into the details (as everything is presented in the book) but only point out the main steps you have to go through.\u003c/p\u003e\n\u003cp dir=\"auto\"\u003eFirst, reinstall your Python dependencies with the AWS group:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry install --with aws\"\u003e\u003cpre\u003epoetry install --with aws\u003c/pre\u003e\u003c/div\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch4 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eAWS SageMaker\u003c/h4\u003e\u003ca id=\"user-content-aws-sagemaker\" class=\"anchor\" aria-label=\"Permalink: AWS SageMaker\" href=\"#aws-sagemaker\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cdiv class=\"markdown-alert markdown-alert-note\" dir=\"auto\"\u003e\u003cp class=\"markdown-alert-title\" dir=\"auto\"\u003e\u003csvg class=\"octicon octicon-info mr-2\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"M0 8a8 8 0 1 1 16 0A8 8 0 0 1 0 8Zm8-6.5a6.5 6.5 0 1 0 0 13 6.5 6.5 0 0 0 0-13ZM6.5 7.75A.75.75 0 0 1 7.25 7h1a.75.75 0 0 1 .75.75v2.75h.25a.75.75 0 0 1 0 1.5h-2a.75.75 0 0 1 0-1.5h.25v-2h-.25a.75.75 0 0 1-.75-.75ZM8 6a1 1 0 1 1 0-2 1 1 0 0 1 0 2Z\"\u003e\u003c/path\u003e\u003c/svg\u003eNote\u003c/p\u003e\u003cp dir=\"auto\"\u003eChapter 10 provides step-by-step instructions in the section \"Implementing the LLM microservice using AWS SageMaker\".\u003c/p\u003e\n\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eBy this point, we expect you to have AWS CLI installed and your AWS CLI and project's env vars (within the \u003ccode\u003e.env\u003c/code\u003e file) properly configured with an AWS admin user.\u003c/p\u003e\n\u003cp dir=\"auto\"\u003eTo ensure best practices, we must create a new AWS user restricted to creating and deleting only resources related to AWS SageMaker. Create it by running:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe create-sagemaker-role\"\u003e\u003cpre\u003epoetry poe create-sagemaker-role\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eIt will create a \u003ccode\u003esagemaker_user_credentials.json\u003c/code\u003e file at the root of your repository with your new \u003ccode\u003eAWS_ACCESS_KEY\u003c/code\u003e and \u003ccode\u003eAWS_SECRET_KEY\u003c/code\u003e values. \u003cstrong\u003eBut before replacing your new AWS credentials, also run the following command to create the execution role (to create it using your admin credentials).\u003c/strong\u003e\u003c/p\u003e\n\u003cp dir=\"auto\"\u003eTo create the IAM execution role used by AWS SageMaker to access other AWS resources on our behalf, run the following:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe create-sagemaker-execution-role\"\u003e\u003cpre\u003epoetry poe create-sagemaker-execution-role\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eIt will create a \u003ccode\u003esagemaker_execution_role.json\u003c/code\u003e file at the root of your repository with your new \u003ccode\u003eAWS_ARN_ROLE\u003c/code\u003e value. Add it to your \u003ccode\u003e.env\u003c/code\u003e file.\u003c/p\u003e\n\u003cp dir=\"auto\"\u003eOnce you've updated the \u003ccode\u003eAWS_ACCESS_KEY\u003c/code\u003e, \u003ccode\u003eAWS_SECRET_KEY\u003c/code\u003e, and \u003ccode\u003eAWS_ARN_ROLE\u003c/code\u003e values in your \u003ccode\u003e.env\u003c/code\u003e file, you can use AWS SageMaker. \u003cstrong\u003eNote that this step is crucial to complete the AWS setup.\u003c/strong\u003e\u003c/p\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch4 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eTraining\u003c/h4\u003e\u003ca id=\"user-content-training\" class=\"anchor\" aria-label=\"Permalink: Training\" href=\"#training\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eWe start the training pipeline through ZenML by running the following:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe run-training-pipeline\"\u003e\u003cpre\u003epoetry poe run-training-pipeline\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eThis will start the training code using the configs from \u003ccode\u003econfigs/training.yaml\u003c/code\u003e directly in SageMaker. You can visualize the results in Comet ML's dashboard.\u003c/p\u003e\n\u003cp dir=\"auto\"\u003eWe start the evaluation pipeline through ZenML by running the following:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe run-evaluation-pipeline\"\u003e\u003cpre\u003epoetry poe run-evaluation-pipeline\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eThis will start the evaluation code using the configs from \u003ccode\u003econfigs/evaluating.yaml\u003c/code\u003e directly in SageMaker. You can visualize the results in \u003ccode\u003e*-results\u003c/code\u003e datasets saved to your Hugging Face profile.\u003c/p\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch4 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eInference\u003c/h4\u003e\u003ca id=\"user-content-inference\" class=\"anchor\" aria-label=\"Permalink: Inference\" href=\"#inference\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eTo create an AWS SageMaker Inference Endpoint, run:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe deploy-inference-endpoint\"\u003e\u003cpre\u003epoetry poe deploy-inference-endpoint\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eTo test it out, run:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe test-sagemaker-endpoint\"\u003e\u003cpre\u003epoetry poe test-sagemaker-endpoint\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eTo delete it, run:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe delete-inference-endpoint\"\u003e\u003cpre\u003epoetry poe delete-inference-endpoint\u003c/pre\u003e\u003c/div\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch4 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eAWS: ML pipelines, artifacts, and containers\u003c/h4\u003e\u003ca id=\"user-content-aws-ml-pipelines-artifacts-and-containers\" class=\"anchor\" aria-label=\"Permalink: AWS: ML pipelines, artifacts, and containers\" href=\"#aws-ml-pipelines-artifacts-and-containers\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eThe ML pipelines, artifacts, and containers are deployed to AWS by leveraging ZenML's deployment features. Thus, you must create an account with ZenML Cloud and follow their guide on deploying a ZenML stack to AWS. Otherwise, we provide step-by-step instructions in \u003cstrong\u003eChapter 11\u003c/strong\u003e, section \u003cstrong\u003eDeploying the LLM Twin's pipelines to the cloud\u003c/strong\u003e on what you must do.\u003c/p\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch4 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eQdrant \u0026amp; MongoDB\u003c/h4\u003e\u003ca id=\"user-content-qdrant--mongodb\" class=\"anchor\" aria-label=\"Permalink: Qdrant \u0026amp; MongoDB\" href=\"#qdrant--mongodb\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eWe leverage Qdrant's and MongoDB's serverless options when deploying the project. Thus, you can either follow \u003ca href=\"https://qdrant.tech/documentation/cloud/create-cluster/\" rel=\"nofollow\"\u003eQdrant's\u003c/a\u003e and \u003ca href=\"https://www.mongodb.com/resources/products/fundamentals/mongodb-cluster-setup\" rel=\"nofollow\"\u003eMongoDB's\u003c/a\u003e tutorials on how to create a freemium cluster for each or go through \u003cstrong\u003eChapter 11\u003c/strong\u003e, section \u003cstrong\u003eDeploying the LLM Twin's pipelines to the cloud\u003c/strong\u003e and follow our step-by-step instructions.\u003c/p\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch4 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eGitHub Actions\u003c/h4\u003e\u003ca id=\"user-content-github-actions\" class=\"anchor\" aria-label=\"Permalink: GitHub Actions\" href=\"#github-actions\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eWe use GitHub Actions to implement our CI/CD pipelines. To implement your own, you have to fork our repository and set the following env vars as Actions secrets in your forked repository:\u003c/p\u003e\n\u003cul dir=\"auto\"\u003e\n\u003cli\u003e\u003ccode\u003eAWS_ACCESS_KEY_ID\u003c/code\u003e\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eAWS_SECRET_ACCESS_KEY\u003c/code\u003e\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eAWS_ECR_NAME\u003c/code\u003e\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eAWS_REGION\u003c/code\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp dir=\"auto\"\u003eAlso, we provide instructions on how to set everything up in \u003cstrong\u003eChapter 11\u003c/strong\u003e, section \u003cstrong\u003eAdding LLMOps to the LLM Twin\u003c/strong\u003e.\u003c/p\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch4 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eComet ML \u0026amp; Opik\u003c/h4\u003e\u003ca id=\"user-content-comet-ml--opik-1\" class=\"anchor\" aria-label=\"Permalink: Comet ML \u0026amp; Opik\" href=\"#comet-ml--opik-1\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eYou can visualize the results on their self-hosted dashboards if you create a Comet account and correctly set the \u003ccode\u003eCOMET_API_KEY\u003c/code\u003e env var. As Opik is powered by Comet, you don't have to set up anything else along Comet:\u003c/p\u003e\n\u003cul dir=\"auto\"\u003e\n\u003cli\u003e\u003ca href=\"https://www.comet.com/?utm_source=llm_handbook\u0026amp;utm_medium=github\u0026amp;utm_campaign=opik\" rel=\"nofollow\"\u003eComet ML (for experiment tracking)\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://www.comet.com/opik?utm_source=llm_handbook\u0026amp;utm_medium=github\u0026amp;utm_campaign=opik\" rel=\"nofollow\"\u003eOpik (for prompt monitoring)\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch3 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003e💰 Running the Project Costs\u003c/h3\u003e\u003ca id=\"user-content--running-the-project-costs\" class=\"anchor\" aria-label=\"Permalink: 💰 Running the Project Costs\" href=\"#-running-the-project-costs\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eWe will mostly stick to free tiers for all the services except for AWS and OpenAI's API, which are both pay-as-you-go services. The cost of running the project once, with our default values, will be roughly ~$25 (most of it comes from using AWS SageMaker for training and inference).\u003c/p\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch2 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003e⚡ Pipelines\u003c/h2\u003e\u003ca id=\"user-content--pipelines\" class=\"anchor\" aria-label=\"Permalink: ⚡ Pipelines\" href=\"#-pipelines\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eAll the ML pipelines will be orchestrated behind the scenes by \u003ca href=\"https://www.zenml.io/\" rel=\"nofollow\"\u003eZenML\u003c/a\u003e. A few exceptions exist when running utility scrips, such as exporting or importing from the data warehouse.\u003c/p\u003e\n\u003cp dir=\"auto\"\u003eThe ZenML pipelines are the entry point for most processes throughout this project. They are under the \u003ccode\u003epipelines/\u003c/code\u003e folder. Thus, when you want to understand or debug a workflow, starting with the ZenML pipeline is the best approach.\u003c/p\u003e\n\u003cp dir=\"auto\"\u003eTo see the pipelines running and their results:\u003c/p\u003e\n\u003cul dir=\"auto\"\u003e\n\u003cli\u003ego to your ZenML dashboard\u003c/li\u003e\n\u003cli\u003ego to the \u003ccode\u003ePipelines\u003c/code\u003e section\u003c/li\u003e\n\u003cli\u003eclick on a specific pipeline (e.g., \u003ccode\u003efeature_engineering\u003c/code\u003e)\u003c/li\u003e\n\u003cli\u003eclick on a specific run (e.g., \u003ccode\u003efeature_engineering_run_2024_06_20_18_40_24\u003c/code\u003e)\u003c/li\u003e\n\u003cli\u003eclick on a specific step or artifact of the DAG to find more details about it\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp dir=\"auto\"\u003eNow, let's explore all the pipelines you can run. From data collection to training, we will present them in their natural order to go through the LLM project end-to-end.\u003c/p\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch3 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eData pipelines\u003c/h3\u003e\u003ca id=\"user-content-data-pipelines\" class=\"anchor\" aria-label=\"Permalink: Data pipelines\" href=\"#data-pipelines\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eRun the data collection ETL:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe run-digital-data-etl\"\u003e\u003cpre\u003epoetry poe run-digital-data-etl\u003c/pre\u003e\u003c/div\u003e\n\u003cdiv class=\"markdown-alert markdown-alert-warning\" dir=\"auto\"\u003e\u003cp class=\"markdown-alert-title\" dir=\"auto\"\u003e\u003csvg class=\"octicon octicon-alert mr-2\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"M6.457 1.047c.659-1.234 2.427-1.234 3.086 0l6.082 11.378A1.75 1.75 0 0 1 14.082 15H1.918a1.75 1.75 0 0 1-1.543-2.575Zm1.763.707a.25.25 0 0 0-.44 0L1.698 13.132a.25.25 0 0 0 .22.368h12.164a.25.25 0 0 0 .22-.368Zm.53 3.996v2.5a.75.75 0 0 1-1.5 0v-2.5a.75.75 0 0 1 1.5 0ZM9 11a1 1 0 1 1-2 0 1 1 0 0 1 2 0Z\"\u003e\u003c/path\u003e\u003c/svg\u003eWarning\u003c/p\u003e\u003cp dir=\"auto\"\u003eYou must have Chrome (or another Chromium-based browser) installed on your system for LinkedIn and Medium crawlers to work (which use Selenium under the hood). Based on your Chrome version, the Chromedriver will be automatically installed to enable Selenium support. Another option is to run everything using our Docker image if you don't want to install Chrome. For example, to run all the pipelines combined you can run \u003ccode\u003epoetry poe run-docker-end-to-end-data-pipeline\u003c/code\u003e. Note that the command can be tweaked to support any other pipeline.\u003c/p\u003e\n\u003cp dir=\"auto\"\u003eIf, for any other reason, you don't have a Chromium-based browser installed and don't want to use Docker, you have two other options to bypass this Selenium issue:\u003c/p\u003e\n\u003cul dir=\"auto\"\u003e\n\u003cli\u003eComment out all the code related to Selenium, Chrome and all the links that use Selenium to crawl them (e.g., Medium), such as the \u003ccode\u003echromedriver_autoinstaller.install()\u003c/code\u003e command from \u003ca href=\"https://github.com/PacktPublishing/LLM-Engineers-Handbook/blob/main/llm_engineering/application/crawlers/base.py\"\u003eapplication.crawlers.base\u003c/a\u003e and other static calls that check for Chrome drivers and Selenium.\u003c/li\u003e\n\u003cli\u003eInstall Google Chrome using your CLI in environments such as GitHub Codespaces or other cloud VMs using the same command as in our \u003ca href=\"https://github.com/PacktPublishing/LLM-Engineers-Handbook/blob/main/Dockerfile#L10\"\u003eDocker file\u003c/a\u003e.\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eTo add additional links to collect from, go to \u003ccode\u003econfigs/digital_data_etl_[author_name].yaml\u003c/code\u003e and add them to the \u003ccode\u003elinks\u003c/code\u003e field. Also, you can create a completely new file and specify it at run time, like this: \u003ccode\u003epython -m llm_engineering.interfaces.orchestrator.run --run-etl --etl-config-filename configs/digital_data_etl_[your_name].yaml\u003c/code\u003e\u003c/p\u003e\n\u003cp dir=\"auto\"\u003eRun the feature engineering pipeline:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe run-feature-engineering-pipeline\"\u003e\u003cpre\u003epoetry poe run-feature-engineering-pipeline\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eGenerate the instruct dataset:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe run-generate-instruct-datasets-pipeline\"\u003e\u003cpre\u003epoetry poe run-generate-instruct-datasets-pipeline\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eGenerate the preference dataset:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe run-generate-preference-datasets-pipeline\"\u003e\u003cpre\u003epoetry poe run-generate-preference-datasets-pipeline\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eRun all of the above compressed into a single pipeline:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe run-end-to-end-data-pipeline\"\u003e\u003cpre\u003epoetry poe run-end-to-end-data-pipeline\u003c/pre\u003e\u003c/div\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch3 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eUtility pipelines\u003c/h3\u003e\u003ca id=\"user-content-utility-pipelines\" class=\"anchor\" aria-label=\"Permalink: Utility pipelines\" href=\"#utility-pipelines\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eExport the data from the data warehouse to JSON files:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe run-export-data-warehouse-to-json\"\u003e\u003cpre\u003epoetry poe run-export-data-warehouse-to-json\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eImport data to the data warehouse from JSON files (by default, it imports the data from the \u003ccode\u003edata/data_warehouse_raw_data\u003c/code\u003e directory):\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe run-import-data-warehouse-from-json\"\u003e\u003cpre\u003epoetry poe run-import-data-warehouse-from-json\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eExport ZenML artifacts to JSON:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe run-export-artifact-to-json-pipeline\"\u003e\u003cpre\u003epoetry poe run-export-artifact-to-json-pipeline\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eThis will export the following ZenML artifacts to the \u003ccode\u003eoutput\u003c/code\u003e folder as JSON files (it will take their latest version):\u003c/p\u003e\n\u003cul dir=\"auto\"\u003e\n\u003cli\u003ecleaned_documents.json\u003c/li\u003e\n\u003cli\u003einstruct_datasets.json\u003c/li\u003e\n\u003cli\u003epreference_datasets.json\u003c/li\u003e\n\u003cli\u003eraw_documents.json\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp dir=\"auto\"\u003eYou can configure what artifacts to export by tweaking the \u003ccode\u003econfigs/export_artifact_to_json.yaml\u003c/code\u003e configuration file.\u003c/p\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch3 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eTraining pipelines\u003c/h3\u003e\u003ca id=\"user-content-training-pipelines\" class=\"anchor\" aria-label=\"Permalink: Training pipelines\" href=\"#training-pipelines\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eRun the training pipeline:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe run-training-pipeline\"\u003e\u003cpre\u003epoetry poe run-training-pipeline\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eRun the evaluation pipeline:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe run-evaluation-pipeline\"\u003e\u003cpre\u003epoetry poe run-evaluation-pipeline\u003c/pre\u003e\u003c/div\u003e\n\u003cdiv class=\"markdown-alert markdown-alert-warning\" dir=\"auto\"\u003e\u003cp class=\"markdown-alert-title\" dir=\"auto\"\u003e\u003csvg class=\"octicon octicon-alert mr-2\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"M6.457 1.047c.659-1.234 2.427-1.234 3.086 0l6.082 11.378A1.75 1.75 0 0 1 14.082 15H1.918a1.75 1.75 0 0 1-1.543-2.575Zm1.763.707a.25.25 0 0 0-.44 0L1.698 13.132a.25.25 0 0 0 .22.368h12.164a.25.25 0 0 0 .22-.368Zm.53 3.996v2.5a.75.75 0 0 1-1.5 0v-2.5a.75.75 0 0 1 1.5 0ZM9 11a1 1 0 1 1-2 0 1 1 0 0 1 2 0Z\"\u003e\u003c/path\u003e\u003c/svg\u003eWarning\u003c/p\u003e\u003cp dir=\"auto\"\u003eFor this to work, make sure you properly configured AWS SageMaker as described in \u003ca href=\"#set-up-cloud-infrastructure-for-production\"\u003eSet up cloud infrastructure (for production)\u003c/a\u003e.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch3 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eInference pipelines\u003c/h3\u003e\u003ca id=\"user-content-inference-pipelines\" class=\"anchor\" aria-label=\"Permalink: Inference pipelines\" href=\"#inference-pipelines\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eCall the RAG retrieval module with a test query:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe call-rag-retrieval-module\"\u003e\u003cpre\u003epoetry poe call-rag-retrieval-module\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eStart the inference real-time RESTful API:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe run-inference-ml-service\"\u003e\u003cpre\u003epoetry poe run-inference-ml-service\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eCall the inference real-time RESTful API with a test query:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe call-inference-ml-service\"\u003e\u003cpre\u003epoetry poe call-inference-ml-service\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eRemember that you can monitor the prompt traces on \u003ca href=\"https://www.comet.com/opik\" rel=\"nofollow\"\u003eOpik\u003c/a\u003e.\u003c/p\u003e\n\u003cdiv class=\"markdown-alert markdown-alert-warning\" dir=\"auto\"\u003e\u003cp class=\"markdown-alert-title\" dir=\"auto\"\u003e\u003csvg class=\"octicon octicon-alert mr-2\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"M6.457 1.047c.659-1.234 2.427-1.234 3.086 0l6.082 11.378A1.75 1.75 0 0 1 14.082 15H1.918a1.75 1.75 0 0 1-1.543-2.575Zm1.763.707a.25.25 0 0 0-.44 0L1.698 13.132a.25.25 0 0 0 .22.368h12.164a.25.25 0 0 0 .22-.368Zm.53 3.996v2.5a.75.75 0 0 1-1.5 0v-2.5a.75.75 0 0 1 1.5 0ZM9 11a1 1 0 1 1-2 0 1 1 0 0 1 2 0Z\"\u003e\u003c/path\u003e\u003c/svg\u003eWarning\u003c/p\u003e\u003cp dir=\"auto\"\u003eFor the inference service to work, you must have the LLM microservice deployed to AWS SageMaker, as explained in the setup cloud infrastructure section.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch3 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eLinting \u0026amp; formatting (QA)\u003c/h3\u003e\u003ca id=\"user-content-linting--formatting-qa\" class=\"anchor\" aria-label=\"Permalink: Linting \u0026amp; formatting (QA)\" href=\"#linting--formatting-qa\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eCheck or fix your linting issues:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe lint-check\npoetry poe lint-fix\"\u003e\u003cpre\u003epoetry poe lint-check\npoetry poe lint-fix\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eCheck or fix your formatting issues:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe format-check\npoetry poe format-fix\"\u003e\u003cpre\u003epoetry poe format-check\npoetry poe format-fix\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eCheck the code for leaked credentials:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe gitleaks-check\"\u003e\u003cpre\u003epoetry poe gitleaks-check\u003c/pre\u003e\u003c/div\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch3 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eTests\u003c/h3\u003e\u003ca id=\"user-content-tests\" class=\"anchor\" aria-label=\"Permalink: Tests\" href=\"#tests\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eRun all the tests using the following command:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe test\"\u003e\u003cpre\u003epoetry poe \u003cspan class=\"pl-c1\"\u003etest\u003c/span\u003e\u003c/pre\u003e\u003c/div\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch2 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003e🏃 Run project\u003c/h2\u003e\u003ca id=\"user-content--run-project\" class=\"anchor\" aria-label=\"Permalink: 🏃 Run project\" href=\"#-run-project\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eBased on the setup and usage steps described above, assuming the local and cloud infrastructure works and the \u003ccode\u003e.env\u003c/code\u003e is filled as expected, follow the next steps to run the LLM system end-to-end:\u003c/p\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch3 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eData\u003c/h3\u003e\u003ca id=\"user-content-data\" class=\"anchor\" aria-label=\"Permalink: Data\" href=\"#data\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003col dir=\"auto\"\u003e\n\u003cli\u003e\n\u003cp dir=\"auto\"\u003eCollect data: \u003ccode\u003epoetry poe run-digital-data-etl\u003c/code\u003e\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp dir=\"auto\"\u003eCompute features: \u003ccode\u003epoetry poe run-feature-engineering-pipeline\u003c/code\u003e\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp dir=\"auto\"\u003eCompute instruct dataset: \u003ccode\u003epoetry poe run-generate-instruct-datasets-pipeline\u003c/code\u003e\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp dir=\"auto\"\u003eCompute preference alignment dataset: \u003ccode\u003epoetry poe run-generate-preference-datasets-pipeline\u003c/code\u003e\u003c/p\u003e\n\u003c/li\u003e\n\u003c/ol\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch3 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eTraining\u003c/h3\u003e\u003ca id=\"user-content-training-1\" class=\"anchor\" aria-label=\"Permalink: Training\" href=\"#training-1\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cdiv class=\"markdown-alert markdown-alert-important\" dir=\"auto\"\u003e\u003cp class=\"markdown-alert-title\" dir=\"auto\"\u003e\u003csvg class=\"octicon octicon-report mr-2\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"M0 1.75C0 .784.784 0 1.75 0h12.5C15.216 0 16 .784 16 1.75v9.5A1.75 1.75 0 0 1 14.25 13H8.06l-2.573 2.573A1.458 1.458 0 0 1 3 14.543V13H1.75A1.75 1.75 0 0 1 0 11.25Zm1.75-.25a.25.25 0 0 0-.25.25v9.5c0 .138.112.25.25.25h2a.75.75 0 0 1 .75.75v2.19l2.72-2.72a.749.749 0 0 1 .53-.22h6.5a.25.25 0 0 0 .25-.25v-9.5a.25.25 0 0 0-.25-.25Zm7 2.25v2.5a.75.75 0 0 1-1.5 0v-2.5a.75.75 0 0 1 1.5 0ZM9 9a1 1 0 1 1-2 0 1 1 0 0 1 2 0Z\"\u003e\u003c/path\u003e\u003c/svg\u003eImportant\u003c/p\u003e\u003cp dir=\"auto\"\u003eFrom now on, for these steps to work, you need to properly set up AWS SageMaker, such as running \u003ccode\u003epoetry install --with aws\u003c/code\u003e and filling in the AWS-related environment variables and configs.\u003c/p\u003e\n\u003c/div\u003e\n\u003col start=\"5\" dir=\"auto\"\u003e\n\u003cli\u003e\n\u003cp dir=\"auto\"\u003eSFT fine-tuning Llamma 3.1: \u003ccode\u003epoetry poe run-training-pipeline\u003c/code\u003e\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp dir=\"auto\"\u003eFor DPO, go to \u003ccode\u003econfigs/training.yaml\u003c/code\u003e, change \u003ccode\u003efinetuning_type\u003c/code\u003e to \u003ccode\u003edpo\u003c/code\u003e, and run \u003ccode\u003epoetry poe run-training-pipeline\u003c/code\u003e again\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp dir=\"auto\"\u003eEvaluate fine-tuned models: \u003ccode\u003epoetry poe run-evaluation-pipeline\u003c/code\u003e\u003c/p\u003e\n\u003c/li\u003e\n\u003c/ol\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch3 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eInference\u003c/h3\u003e\u003ca id=\"user-content-inference-1\" class=\"anchor\" aria-label=\"Permalink: Inference\" href=\"#inference-1\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cdiv class=\"markdown-alert markdown-alert-important\" dir=\"auto\"\u003e\u003cp class=\"markdown-alert-title\" dir=\"auto\"\u003e\u003csvg class=\"octicon octicon-report mr-2\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"M0 1.75C0 .784.784 0 1.75 0h12.5C15.216 0 16 .784 16 1.75v9.5A1.75 1.75 0 0 1 14.25 13H8.06l-2.573 2.573A1.458 1.458 0 0 1 3 14.543V13H1.75A1.75 1.75 0 0 1 0 11.25Zm1.75-.25a.25.25 0 0 0-.25.25v9.5c0 .138.112.25.25.25h2a.75.75 0 0 1 .75.75v2.19l2.72-2.72a.749.749 0 0 1 .53-.22h6.5a.25.25 0 0 0 .25-.25v-9.5a.25.25 0 0 0-.25-.25Zm7 2.25v2.5a.75.75 0 0 1-1.5 0v-2.5a.75.75 0 0 1 1.5 0ZM9 9a1 1 0 1 1-2 0 1 1 0 0 1 2 0Z\"\u003e\u003c/path\u003e\u003c/svg\u003eImportant\u003c/p\u003e\u003cp dir=\"auto\"\u003eFrom now on, for these steps to work, you need to properly set up AWS SageMaker, such as running \u003ccode\u003epoetry install --with aws\u003c/code\u003e and filling in the AWS-related environment variables and configs.\u003c/p\u003e\n\u003c/div\u003e\n\u003col start=\"8\" dir=\"auto\"\u003e\n\u003cli\u003e\n\u003cp dir=\"auto\"\u003eCall only the RAG retrieval module: \u003ccode\u003epoetry poe call-rag-retrieval-module\u003c/code\u003e\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp dir=\"auto\"\u003eDeploy the LLM Twin microservice to SageMaker: \u003ccode\u003epoetry poe deploy-inference-endpoint\u003c/code\u003e\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp dir=\"auto\"\u003eTest the LLM Twin microservice: \u003ccode\u003epoetry poe test-sagemaker-endpoint\u003c/code\u003e\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp dir=\"auto\"\u003eStart end-to-end RAG server: \u003ccode\u003epoetry poe run-inference-ml-service\u003c/code\u003e\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp dir=\"auto\"\u003eTest RAG server: \u003ccode\u003epoetry poe call-inference-ml-service\u003c/code\u003e\u003c/p\u003e\n\u003c/li\u003e\n\u003c/ol\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch2 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003e📄 License\u003c/h2\u003e\u003ca id=\"user-content--license\" class=\"anchor\" aria-label=\"Permalink: 📄 License\" href=\"#-license\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eThis course is an open-source project released under the MIT license. Thus, as long you distribute our LICENSE and acknowledge our work, you can safely clone or fork this project and use it as a source of inspiration for whatever you want (e.g., university projects, college degree projects, personal projects, etc.).\u003c/p\u003e\n\u003c/article\u003e","loaded":true,"timedOut":false,"errorMessage":null,"headerInfo":{"toc":[{"level":1,"text":"👷 LLM Engineer's Handbook","anchor":"-llm-engineers-handbook","htmlText":"👷 LLM Engineer's Handbook"},{"level":2,"text":"🌟 Features","anchor":"-features","htmlText":"🌟 Features"},{"level":2,"text":"🔗 Dependencies","anchor":"-dependencies","htmlText":"🔗 Dependencies"},{"level":3,"text":"Local dependencies","anchor":"local-dependencies","htmlText":"Local dependencies"},{"level":3,"text":"Cloud services","anchor":"cloud-services","htmlText":"Cloud services"},{"level":2,"text":"🗂️ Project Structure","anchor":"️-project-structure","htmlText":"🗂️ Project Structure"},{"level":2,"text":"💻 Installation","anchor":"-installation","htmlText":"💻 Installation"},{"level":3,"text":"1. Clone the Repository","anchor":"1-clone-the-repository","htmlText":"1. Clone the Repository"},{"level":3,"text":"2. Set Up Python Environment","anchor":"2-set-up-python-environment","htmlText":"2. Set Up Python Environment"},{"level":4,"text":"Option A: Using Global Python (if version 3.11 is installed)","anchor":"option-a-using-global-python-if-version-311-is-installed","htmlText":"Option A: Using Global Python (if version 3.11 is installed)"},{"level":4,"text":"Option B: Using pyenv (recommended)","anchor":"option-b-using-pyenv-recommended","htmlText":"Option B: Using pyenv (recommended)"},{"level":3,"text":"3. Install Dependencies","anchor":"3-install-dependencies","htmlText":"3. Install Dependencies"},{"level":3,"text":"4. Activate the Environment","anchor":"4-activate-the-environment","htmlText":"4. Activate the Environment"},{"level":3,"text":"Alternative Command Execution","anchor":"alternative-command-execution","htmlText":"Alternative Command Execution"},{"level":4,"text":"Example:","anchor":"example","htmlText":"Example:"},{"level":3,"text":"5. Local Development Setup","anchor":"5-local-development-setup","htmlText":"5. Local Development Setup"},{"level":4,"text":"OpenAI","anchor":"openai","htmlText":"OpenAI"},{"level":4,"text":"Hugging Face","anchor":"hugging-face","htmlText":"Hugging Face"},{"level":4,"text":"Comet ML \u0026 Opik","anchor":"comet-ml--opik","htmlText":"Comet ML \u0026amp; Opik"},{"level":3,"text":"6. Deployment Setup","anchor":"6-deployment-setup","htmlText":"6. Deployment Setup"},{"level":4,"text":"MongoDB","anchor":"mongodb","htmlText":"MongoDB"},{"level":4,"text":"Qdrant","anchor":"qdrant","htmlText":"Qdrant"},{"level":4,"text":"AWS","anchor":"aws","htmlText":"AWS"},{"level":2,"text":"🏗️ Infrastructure","anchor":"️-infrastructure","htmlText":"🏗️ Infrastructure"},{"level":3,"text":"Local infrastructure (for testing and development)","anchor":"local-infrastructure-for-testing-and-development","htmlText":"Local infrastructure (for testing and development)"},{"level":4,"text":"ZenML","anchor":"zenml","htmlText":"ZenML"},{"level":4,"text":"Qdrant","anchor":"qdrant-1","htmlText":"Qdrant"},{"level":4,"text":"MongoDB","anchor":"mongodb-1","htmlText":"MongoDB"},{"level":3,"text":"Cloud infrastructure (for production)","anchor":"cloud-infrastructure-for-production","htmlText":"Cloud infrastructure (for production)"},{"level":4,"text":"AWS SageMaker","anchor":"aws-sagemaker","htmlText":"AWS SageMaker"},{"level":4,"text":"Training","anchor":"training","htmlText":"Training"},{"level":4,"text":"Inference","anchor":"inference","htmlText":"Inference"},{"level":4,"text":"AWS: ML pipelines, artifacts, and containers","anchor":"aws-ml-pipelines-artifacts-and-containers","htmlText":"AWS: ML pipelines, artifacts, and containers"},{"level":4,"text":"Qdrant \u0026 MongoDB","anchor":"qdrant--mongodb","htmlText":"Qdrant \u0026amp; MongoDB"},{"level":4,"text":"GitHub Actions","anchor":"github-actions","htmlText":"GitHub Actions"},{"level":4,"text":"Comet ML \u0026 Opik","anchor":"comet-ml--opik-1","htmlText":"Comet ML \u0026amp; Opik"},{"level":3,"text":"💰 Running the Project Costs","anchor":"-running-the-project-costs","htmlText":"💰 Running the Project Costs"},{"level":2,"text":"⚡ Pipelines","anchor":"-pipelines","htmlText":"⚡ Pipelines"},{"level":3,"text":"Data pipelines","anchor":"data-pipelines","htmlText":"Data pipelines"},{"level":3,"text":"Utility pipelines","anchor":"utility-pipelines","htmlText":"Utility pipelines"},{"level":3,"text":"Training pipelines","anchor":"training-pipelines","htmlText":"Training pipelines"},{"level":3,"text":"Inference pipelines","anchor":"inference-pipelines","htmlText":"Inference pipelines"},{"level":3,"text":"Linting \u0026 formatting (QA)","anchor":"linting--formatting-qa","htmlText":"Linting \u0026amp; formatting (QA)"},{"level":3,"text":"Tests","anchor":"tests","htmlText":"Tests"},{"level":2,"text":"🏃 Run project","anchor":"-run-project","htmlText":"🏃 Run project"},{"level":3,"text":"Data","anchor":"data","htmlText":"Data"},{"level":3,"text":"Training","anchor":"training-1","htmlText":"Training"},{"level":3,"text":"Inference","anchor":"inference-1","htmlText":"Inference"},{"level":2,"text":"📄 License","anchor":"-license","htmlText":"📄 License"}],"siteNavLoginPath":"/login?return_to=https%3A%2F%2Fgithub.com%2FPacktPublishing%2FLLM-Engineers-Handbook"}},{"displayName":"LICENSE","repoName":"LLM-Engineers-Handbook","refName":"main","path":"LICENSE","preferredFileType":"license","tabName":"MIT","richText":null,"loaded":false,"timedOut":false,"errorMessage":null,"headerInfo":{"toc":null,"siteNavLoginPath":"/login?return_to=https%3A%2F%2Fgithub.com%2FPacktPublishing%2FLLM-Engineers-Handbook"}}],"overviewFilesProcessingTime":0,"copilotSWEAgentEnabled":false}},"codeViewLayoutRoute":{"repo":{"id":784183059,"defaultBranch":"main","name":"LLM-Engineers-Handbook","ownerLogin":"PacktPublishing","currentUserCanPush":false,"isFork":false,"isEmpty":false,"createdAt":"2024-04-09T10:56:03.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/10974906?v=4","public":true,"private":false,"isOrgOwned":true},"currentUser":null,"uploadToken":"WOxTsk3ONpA8HId0LQWoFVtegXtQklU5KzEGakQRGMxXRLo7_A_KPj7Eimi2PcCycFiWK0I1c3cp84qYaQq9Tw","allShortcutsEnabled":false,"treeExpanded":true,"path":"/","symbolsExpanded":false,"refInfo":{"name":"main","listCacheKey":"v0:1735803499.0","canEdit":false,"currentOid":"e6038da300616020695f8effb3cbf0a78aac5870"},"helpUrl":"https://docs.github.com","findFileWorkerPath":"/assets-cdn/worker/find-file-worker-4e5d7136862a2a48.js","findInFileWorkerPath":"/assets-cdn/worker/find-in-file-worker-4c35b25d88167fef.js","githubDevUrl":null},"csrf_tokens":{"/PacktPublishing/LLM-Engineers-Handbook/branches":{"post":"7hAIiPebH9R5AgJNxe3XQbueE84wKOQ4HeVTs8e6prglNM2QtHf5NgDRXFX1F2ymEHBf4I-KhYXuoy-j_l_seg"}}},"title":"GitHub - PacktPublishing/LLM-Engineers-Handbook: The LLM's practical guide: From the fundamentals to deploying advanced LLM and RAG apps to AWS using LLMOps best practices","appPayload":{},"meta":{"title":"GitHub - PacktPublishing/LLM-Engineers-Handbook: The LLM's practical guide: From the fundamentals to deploying advanced LLM and RAG apps to AWS using LLMOps best practices"}} PacktPublishing/LLM-Engineers-Handbook mainBranchesTagsGo to fileCodeOpen more actions menuFolders and filesNameNameLast commit messageLast commit dateLatest commit History201 Commits201 Commits.github/workflows.github/workflows .vscode.vscode code_snippetscode_snippets configsconfigs datadata imagesimages llm_engineeringllm_engineering pipelinespipelines stepssteps teststests toolstools .env.example.env.example .gitignore.gitignore .pre-commit-config.yaml.pre-commit-config.yaml .python-version.python-version DockerfileDockerfile LICENSELICENSE README.mdREADME.md docker-compose.ymldocker-compose.yml poetry.lockpoetry.lock pyproject.tomlpyproject.toml ruff.tomlruff.toml View all filesRepository files navigationREADMEMIT license 👷 LLM Engineer's Handbook Official repository of the LLM Engineer's Handbook by Paul Iusztin and Maxime Labonne Find the book on Amazon or Packt 🌟 Features The goal of this book is to create your own end-to-end LLM-based system using best practices: 📝 Data collection & generation 🔄 LLM training pipeline 📊 Simple RAG system 🚀 Production-ready AWS deployment 🔍 Comprehensive monitoring 🧪 Testing and evaluation framework You can download and use the final trained model on Hugging Face. ImportantThe code in this GitHub repository is actively maintained and may contain updates not reflected in the book. Always refer to this repository for the latest version of the code. 🔗 Dependencies Local dependencies To install and run the project locally, you need the following dependencies. Tool Version Purpose Installation Link pyenv ≥2.3.36 Multiple Python versions (optional) Install Guide Python 3.11 Runtime environment Download Poetry >= 1.8.3 and < 2.0 Package management Install Guide Docker ≥27.1.1 Containerization Install Guide AWS CLI ≥2.15.42 Cloud management Install Guide Git ≥2.44.0 Version control Download Cloud services The code also uses and depends on the following cloud services. For now, you don't have to do anything. We will guide you in the installation and deployment sections on how to use them: Service Purpose HuggingFace Model registry Comet ML Experiment tracker Opik Prompt monitoring ZenML Orchestrator and artifacts layer AWS Compute and storage MongoDB NoSQL database Qdrant Vector database GitHub Actions CI/CD pipeline In the LLM Engineer's Handbook, Chapter 2 will walk you through each tool. Chapters 10 and 11 provide step-by-step guides on how to set up everything you need. 🗂️ Project Structure Here is the directory overview: . ├── code_snippets/ # Standalone example code ├── configs/ # Pipeline configuration files ├── llm_engineering/ # Core project package │ ├── application/ │ ├── domain/ │ ├── infrastructure/ │ ├── model/ ├── pipelines/ # ML pipeline definitions ├── steps/ # Pipeline components ├── tests/ # Test examples ├── tools/ # Utility scripts │ ├── run.py │ ├── ml_service.py │ ├── rag.py │ ├── data_warehouse.py llm_engineering/ is the main Python package implementing LLM and RAG functionality. It follows Domain-Driven Design (DDD) principles: domain/: Core business entities and structures application/: Business logic, crawlers, and RAG implementation model/: LLM training and inference infrastructure/: External service integrations (AWS, Qdrant, MongoDB, FastAPI) The code logic and imports flow as follows: infrastructure → model → application → domain pipelines/: Contains the ZenML ML pipelines, which serve as the entry point for all the ML pipelines. Coordinates the data processing and model training stages of the ML lifecycle. steps/: Contains individual ZenML steps, which are reusable components for building and customizing ZenML pipelines. Steps perform specific tasks (e.g., data loading, preprocessing) and can be combined within the ML pipelines. tests/: Covers a few sample tests used as examples within the CI pipeline. tools/: Utility scripts used to call the ZenML pipelines and inference code: run.py: Entry point script to run ZenML pipelines. ml_service.py: Starts the REST API inference server. rag.py: Demonstrates usage of the RAG retrieval module. data_warehouse.py: Used to export or import data from the MongoDB data warehouse through JSON files. configs/: ZenML YAML configuration files to control the execution of pipelines and steps. code_snippets/: Independent code examples that can be executed independently. 💻 Installation NoteIf you are experiencing issues while installing and running the repository, consider checking the Issues GitHub section for other people who solved similar problems or directly asking us for help. 1. Clone the Repository Start by cloning the repository and navigating to the project directory: git clone https://github.com/PacktPublishing/LLM-Engineers-Handbook.git cd LLM-Engineers-Handbook Next, we have to prepare your Python environment and its adjacent dependencies. 2. Set Up Python Environment The project requires Python 3.11. You can either use your global Python installation or set up a project-specific version using pyenv. Option A: Using Global Python (if version 3.11 is installed) Verify your Python version: python --version # Should show Python 3.11.x Option B: Using pyenv (recommended) Verify pyenv installation: pyenv --version # Should show pyenv 2.3.36 or later Install Python 3.11.8: pyenv install 3.11.8 Verify the installation: python --version # Should show Python 3.11.8 Confirm Python version in the project directory: python --version # Output: Python 3.11.8 NoteThe project includes a .python-version file that automatically sets the correct Python version when you're in the project directory. 3. Install Dependencies The project uses Poetry for dependency management. Verify Poetry installation: poetry --version # Should show Poetry version 1.8.3 or later Set up the project environment and install dependencies: poetry env use 3.11 poetry install --without aws poetry run pre-commit install This will: Configure Poetry to use Python 3.11 Install project dependencies (excluding AWS-specific packages) Set up pre-commit hooks for code verification 4. Activate the Environment As our task manager, we run all the scripts using Poe the Poet. Start a Poetry shell: poetry shell Run project commands using Poe the Poet: poetry poe ... 🔧 Troubleshooting Poe the Poet Installation Alternative Command Execution If you're experiencing issues with poethepoet, you can still run the project commands directly through Poetry. Here's how: Look up the command definition in pyproject.toml Use poetry run with the underlying command Example: Instead of: poetry poe local-infrastructure-up Use the direct command from pyproject.toml: poetry run <actual-command-from-pyproject-toml> Note: All project commands are defined in the [tool.poe.tasks] section of pyproject.toml Now, let's configure our local project with all the necessary credentials and tokens to run the code locally. 5. Local Development Setup After you have installed all the dependencies, you must create and fill a .env file with your credentials to appropriately interact with other services and run the project. Setting your sensitive credentials in a .env file is a good security practice, as this file won't be committed to GitHub or shared with anyone else. First, copy our example by running the following: cp .env.example .env # The file must be at your repository's root! Now, let's understand how to fill in all the essential variables within the .env file to get you started. The following are the mandatory settings we must complete when working locally: OpenAI To authenticate to OpenAI's API, you must fill out the OPENAI_API_KEY env var with an authentication token. OPENAI_API_KEY=your_api_key_here → Check out this tutorial to learn how to provide one from OpenAI. Hugging Face To authenticate to Hugging Face, you must fill out the HUGGINGFACE_ACCESS_TOKEN env var with an authentication token. HUGGINGFACE_ACCESS_TOKEN=your_token_here → Check out this tutorial to learn how to provide one from Hugging Face. Comet ML & Opik To authenticate to Comet ML (required only during training) and Opik, you must fill out the COMET_API_KEY env var with your authentication token. COMET_API_KEY=your_api_key_here → Check out this tutorial to learn how to get started with Opik. You can also access Opik's dashboard using 🔗this link. 6. Deployment Setup When deploying the project to the cloud, we must set additional settings for Mongo, Qdrant, and AWS. If you are just working locally, the default values of these env vars will work out of the box. Detailed deployment instructions are available in Chapter 11 of the LLM Engineer's Handbook. MongoDB We must change the DATABASE_HOST env var with the URL pointing to your cloud MongoDB cluster. DATABASE_HOST=your_mongodb_url → Check out this tutorial to learn how to create and host a MongoDB cluster for free. Qdrant Change USE_QDRANT_CLOUD to true, QDRANT_CLOUD_URL with the URL point to your cloud Qdrant cluster, and QDRANT_APIKEY with its API key. USE_QDRANT_CLOUD=true QDRANT_CLOUD_URL=your_qdrant_cloud_url QDRANT_APIKEY=your_qdrant_api_key → Check out this tutorial to learn how to create a Qdrant cluster for free AWS For your AWS set-up to work correctly, you need the AWS CLI installed on your local machine and properly configured with an admin user (or a user with enough permissions to create new SageMaker, ECR, and S3 resources; using an admin user will make everything more straightforward). Chapter 2 provides step-by-step instructions on how to install the AWS CLI, create an admin user on AWS, and get an access key to set up the AWS_ACCESS_KEY and AWS_SECRET_KEY environment variables. If you already have an AWS admin user in place, you have to configure the following env vars in your .env file: AWS_REGION=eu-central-1 # Change it with your AWS region. AWS_ACCESS_KEY=your_aws_access_key AWS_SECRET_KEY=your_aws_secret_key AWS credentials are typically stored in ~/.aws/credentials. You can view this file directly using cat or similar commands: cat ~/.aws/credentials ImportantAdditional configuration options are available in settings.py. Any variable in the Settings class can be configured through the .env file. 🏗️ Infrastructure Local infrastructure (for testing and development) When running the project locally, we host a MongoDB and Qdrant database using Docker. Also, a testing ZenML server is made available through their Python package. WarningYou need Docker installed (>= v27.1.1) For ease of use, you can start the whole local development infrastructure with the following command: poetry poe local-infrastructure-up Also, you can stop the ZenML server and all the Docker containers using the following command: poetry poe local-infrastructure-down WarningWhen running on MacOS, before starting the server, export the following environment variable: export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES Otherwise, the connection between the local server and pipeline will break. 🔗 More details in this issue. This is done by default when using Poe the Poet. Start the inference real-time RESTful API: poetry poe run-inference-ml-service ImportantThe LLM microservice, called by the RESTful API, will work only after deploying the LLM to AWS SageMaker. ZenML Dashboard URL: localhost:8237 Default credentials: username: default password: → Find out more about using and setting up ZenML. Qdrant REST API URL: localhost:6333 Dashboard URL: localhost:6333/dashboard → Find out more about using and setting up Qdrant with Docker. MongoDB Database URI: mongodb://llm_engineering:[email protected]:27017 Database name: twin Default credentials: username: llm_engineering password: llm_engineering → Find out more about using and setting up MongoDB with Docker. You can search your MongoDB collections using your IDEs MongoDB plugin (which you have to install separately), where you have to use the database URI to connect to the MongoDB database hosted within the Docker container: mongodb://llm_engineering:[email protected]:27017 ImportantEverything related to training or running the LLMs (e.g., training, evaluation, inference) can only be run if you set up AWS SageMaker, as explained in the next section on cloud infrastructure. Cloud infrastructure (for production) Here we will quickly present how to deploy the project to AWS and other serverless services. We won't go into the details (as everything is presented in the book) but only point out the main steps you have to go through. First, reinstall your Python dependencies with the AWS group: poetry install --with aws AWS SageMaker NoteChapter 10 provides step-by-step instructions in the section "Implementing the LLM microservice using AWS SageMaker". By this point, we expect you to have AWS CLI installed and your AWS CLI and project's env vars (within the .env file) properly configured with an AWS admin user. To ensure best practices, we must create a new AWS user restricted to creating and deleting only resources related to AWS SageMaker. Create it by running: poetry poe create-sagemaker-role It will create a sagemaker_user_credentials.json file at the root of your repository with your new AWS_ACCESS_KEY and AWS_SECRET_KEY values. But before replacing your new AWS credentials, also run the following command to create the execution role (to create it using your admin credentials). To create the IAM execution role used by AWS SageMaker to access other AWS resources on our behalf, run the following: poetry poe create-sagemaker-execution-role It will create a sagemaker_execution_role.json file at the root of your repository with your new AWS_ARN_ROLE value. Add it to your .env file. Once you've updated the AWS_ACCESS_KEY, AWS_SECRET_KEY, and AWS_ARN_ROLE values in your .env file, you can use AWS SageMaker. Note that this step is crucial to complete the AWS setup. Training We start the training pipeline through ZenML by running the following: poetry poe run-training-pipeline This will start the training code using the configs from configs/training.yaml directly in SageMaker. You can visualize the results in Comet ML's dashboard. We start the evaluation pipeline through ZenML by running the following: poetry poe run-evaluation-pipeline This will start the evaluation code using the configs from configs/evaluating.yaml directly in SageMaker. You can visualize the results in *-results datasets saved to your Hugging Face profile. Inference To create an AWS SageMaker Inference Endpoint, run: poetry poe deploy-inference-endpoint To test it out, run: poetry poe test-sagemaker-endpoint To delete it, run: poetry poe delete-inference-endpoint AWS: ML pipelines, artifacts, and containers The ML pipelines, artifacts, and containers are deployed to AWS by leveraging ZenML's deployment features. Thus, you must create an account with ZenML Cloud and follow their guide on deploying a ZenML stack to AWS. Otherwise, we provide step-by-step instructions in Chapter 11, section Deploying the LLM Twin's pipelines to the cloud on what you must do. Qdrant & MongoDB We leverage Qdrant's and MongoDB's serverless options when deploying the project. Thus, you can either follow Qdrant's and MongoDB's tutorials on how to create a freemium cluster for each or go through Chapter 11, section Deploying the LLM Twin's pipelines to the cloud and follow our step-by-step instructions. GitHub Actions We use GitHub Actions to implement our CI/CD pipelines. To implement your own, you have to fork our repository and set the following env vars as Actions secrets in your forked repository: AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_ECR_NAME AWS_REGION Also, we provide instructions on how to set everything up in Chapter 11, section Adding LLMOps to the LLM Twin. Comet ML & Opik You can visualize the results on their self-hosted dashboards if you create a Comet account and correctly set the COMET_API_KEY env var. As Opik is powered by Comet, you don't have to set up anything else along Comet: Comet ML (for experiment tracking) Opik (for prompt monitoring) 💰 Running the Project Costs We will mostly stick to free tiers for all the services except for AWS and OpenAI's API, which are both pay-as-you-go services. The cost of running the project once, with our default values, will be roughly ~$25 (most of it comes from using AWS SageMaker for training and inference). ⚡ Pipelines All the ML pipelines will be orchestrated behind the scenes by ZenML. A few exceptions exist when running utility scrips, such as exporting or importing from the data warehouse. The ZenML pipelines are the entry point for most processes throughout this project. They are under the pipelines/ folder. Thus, when you want to understand or debug a workflow, starting with the ZenML pipeline is the best approach. To see the pipelines running and their results: go to your ZenML dashboard go to the Pipelines section click on a specific pipeline (e.g., feature_engineering) click on a specific run (e.g., feature_engineering_run_2024_06_20_18_40_24) click on a specific step or artifact of the DAG to find more details about it Now, let's explore all the pipelines you can run. From data collection to training, we will present them in their natural order to go through the LLM project end-to-end. Data pipelines Run the data collection ETL: poetry poe run-digital-data-etl WarningYou must have Chrome (or another Chromium-based browser) installed on your system for LinkedIn and Medium crawlers to work (which use Selenium under the hood). Based on your Chrome version, the Chromedriver will be automatically installed to enable Selenium support. Another option is to run everything using our Docker image if you don't want to install Chrome. For example, to run all the pipelines combined you can run poetry poe run-docker-end-to-end-data-pipeline. Note that the command can be tweaked to support any other pipeline. If, for any other reason, you don't have a Chromium-based browser installed and don't want to use Docker, you have two other options to bypass this Selenium issue: Comment out all the code related to Selenium, Chrome and all the links that use Selenium to crawl them (e.g., Medium), such as the chromedriver_autoinstaller.install() command from application.crawlers.base and other static calls that check for Chrome drivers and Selenium. Install Google Chrome using your CLI in environments such as GitHub Codespaces or other cloud VMs using the same command as in our Docker file. To add additional links to collect from, go to configs/digital_data_etl_[author_name].yaml and add them to the links field. Also, you can create a completely new file and specify it at run time, like this: python -m llm_engineering.interfaces.orchestrator.run --run-etl --etl-config-filename configs/digital_data_etl_[your_name].yaml Run the feature engineering pipeline: poetry poe run-feature-engineering-pipeline Generate the instruct dataset: poetry poe run-generate-instruct-datasets-pipeline Generate the preference dataset: poetry poe run-generate-preference-datasets-pipeline Run all of the above compressed into a single pipeline: poetry poe run-end-to-end-data-pipeline Utility pipelines Export the data from the data warehouse to JSON files: poetry poe run-export-data-warehouse-to-json Import data to the data warehouse from JSON files (by default, it imports the data from the data/data_warehouse_raw_data directory): poetry poe run-import-data-warehouse-from-json Export ZenML artifacts to JSON: poetry poe run-export-artifact-to-json-pipeline This will export the following ZenML artifacts to the output folder as JSON files (it will take their latest version): cleaned_documents.json instruct_datasets.json preference_datasets.json raw_documents.json You can configure what artifacts to export by tweaking the configs/export_artifact_to_json.yaml configuration file. Training pipelines Run the training pipeline: poetry poe run-training-pipeline Run the evaluation pipeline: poetry poe run-evaluation-pipeline WarningFor this to work, make sure you properly configured AWS SageMaker as described in Set up cloud infrastructure (for production). Inference pipelines Call the RAG retrieval module with a test query: poetry poe call-rag-retrieval-module Start the inference real-time RESTful API: poetry poe run-inference-ml-service Call the inference real-time RESTful API with a test query: poetry poe call-inference-ml-service Remember that you can monitor the prompt traces on Opik. WarningFor the inference service to work, you must have the LLM microservice deployed to AWS SageMaker, as explained in the setup cloud infrastructure section. Linting & formatting (QA) Check or fix your linting issues: poetry poe lint-check poetry poe lint-fix Check or fix your formatting issues: poetry poe format-check poetry poe format-fix Check the code for leaked credentials: poetry poe gitleaks-check Tests Run all the tests using the following command: poetry poe test 🏃 Run project Based on the setup and usage steps described above, assuming the local and cloud infrastructure works and the .env is filled as expected, follow the next steps to run the LLM system end-to-end: Data Collect data: poetry poe run-digital-data-etl Compute features: poetry poe run-feature-engineering-pipeline Compute instruct dataset: poetry poe run-generate-instruct-datasets-pipeline Compute preference alignment dataset: poetry poe run-generate-preference-datasets-pipeline Training ImportantFrom now on, for these steps to work, you need to properly set up AWS SageMaker, such as running poetry install --with aws and filling in the AWS-related environment variables and configs. SFT fine-tuning Llamma 3.1: poetry poe run-training-pipeline For DPO, go to configs/training.yaml, change finetuning_type to dpo, and run poetry poe run-training-pipeline again Evaluate fine-tuned models: poetry poe run-evaluation-pipeline Inference ImportantFrom now on, for these steps to work, you need to properly set up AWS SageMaker, such as running poetry install --with aws and filling in the AWS-related environment variables and configs. Call only the RAG retrieval module: poetry poe call-rag-retrieval-module Deploy the LLM Twin microservice to SageMaker: poetry poe deploy-inference-endpoint Test the LLM Twin microservice: poetry poe test-sagemaker-endpoint Start end-to-end RAG server: poetry poe run-inference-ml-service Test RAG server: poetry poe call-inference-ml-service 📄 License This course is an open-source project released under the MIT license. Thus, as long you distribute our LICENSE and acknowledge our work, you can safely clone or fork this project and use it as a source of inspiration for whatever you want (e.g., university projects, college degree projects, personal projects, etc.). About The LLM's practical guide: From the fundamentals to deploying advanced LLM and RAG apps to AWS using LLMOps best practices www.amazon.com/LLM-Engineers-Handbook-engineering-production/dp/1836200072/ Topics aws rag mlops llm llmops genai fine-tuning-llm llm-evaluation ml-system-design Resources Readme License MIT license Uh oh! There was an error while loading. Please reload this page. Activity Custom properties Stars 4.9k stars Watchers 57 watching Forks 1.2k forks Report repository Releases No releases published Packages 0 Uh oh! There was an error while loading. Please reload this page. Uh oh! There was an error while loading. Please reload this page. Contributors Uh oh! There was an error while loading. Please reload this page. Languages Python 99.2% Dockerfile 0.8% {"resolvedServerColorMode":"day"} Footer © 2026 GitHub, Inc. Footer navigation Terms Privacy Security Status Community Docs Contact Manage cookies Do not share my personal information You can’t perform that action at this time.
中文
:root { --tab-size-preference: 4; } pre, code { tab-size: var(--tab-size-preference); } {"locale":"en","featureFlags":["actions_custom_images_storage_billing_ui_visibility","actions_image_version_event","actions_service_container_command","agent_conflict_resolution","alternate_user_config_repo","arianotify_comprehensive_migration","batch_suggested_changes","billing_discount_threshold_notification","code_scanning_alert_tracking_links_phase_2","code_scanning_dfa_degraded_experience_notice","codespaces_prebuild_region_target_update","codespaces_tab_react","coding_agent_model_selection","coding_agent_model_selection_all_skus","coding_agent_third_party_model_ui","comment_viewer_copy_raw_markdown","contentful_primer_code_blocks","copilot_agent_image_upload","copilot_agent_snippy","copilot_api_agentic_issue_marshal_yaml","copilot_ask_mode_dropdown","copilot_chat_attach_multiple_images","copilot_chat_clear_model_selection_for_default_change","copilot_chat_enable_tool_call_logs","copilot_chat_explain_error_user_model","copilot_chat_file_redirect","copilot_chat_input_commands","copilot_chat_opening_thread_switch","copilot_chat_reduce_quota_checks","copilot_chat_search_bar_redirect","copilot_chat_selection_attachments","copilot_chat_vision_in_claude","copilot_chat_vision_preview_gate","copilot_custom_copilots","copilot_custom_copilots_feature_preview","copilot_diff_explain_conversation_intent","copilot_diff_reference_context","copilot_duplicate_thread","copilot_extensions_hide_in_dotcom_chat","copilot_extensions_removal_on_marketplace","copilot_features_sql_server_logo","copilot_file_block_ref_matching","copilot_ftp_hyperspace_upgrade_prompt","copilot_icebreakers_experiment_dashboard","copilot_icebreakers_experiment_hyperspace","copilot_immersive_code_block_transition_wrap","copilot_immersive_embedded","copilot_immersive_file_block_transition_open","copilot_immersive_file_preview_keep_mounted","copilot_immersive_job_result_preview","copilot_immersive_layout_routes","copilot_immersive_structured_model_picker","copilot_immersive_task_hyperlinking","copilot_immersive_task_within_chat_thread","copilot_mc_cli_resume_any_users_task","copilot_mission_control_always_send_integration_id","copilot_mission_control_cli_resume_with_task_id","copilot_mission_control_initial_data_spinner","copilot_mission_control_lazy_load_pr_data","copilot_mission_control_scroll_to_bottom_button","copilot_mission_control_task_alive_updates","copilot_org_policy_page_focus_mode","copilot_redirect_header_button_to_agents","copilot_resource_panel","copilot_scroll_preview_tabs","copilot_share_active_subthread","copilot_spaces_ga","copilot_spaces_individual_policies_ga","copilot_spaces_pagination","copilot_spark_empty_state","copilot_spark_handle_nil_friendly_name","copilot_swe_agent_hide_model_picker_if_only_auto","copilot_swe_agent_pr_comment_model_picker","copilot_swe_agent_use_subagents","copilot_task_api_github_rest_style","copilot_unconfigured_is_inherited","copilot_usage_metrics_ga","copilot_workbench_slim_line_top_tabs","custom_instructions_file_references","dashboard_indexeddb_caching","dashboard_lists_max_age_filter","dashboard_universe_2025_feedback_dialog","flex_cta_groups_mvp","global_nav_react","hyperspace_2025_logged_out_batch_1","hyperspace_2025_logged_out_batch_2","hyperspace_2025_logged_out_batch_3","ipm_global_transactional_message_agents","ipm_global_transactional_message_copilot","ipm_global_transactional_message_issues","ipm_global_transactional_message_prs","ipm_global_transactional_message_repos","ipm_global_transactional_message_spaces","issue_cca_modal_open","issue_cca_visualization","issue_fields_global_search","issues_expanded_file_types","issues_lazy_load_comment_box_suggestions","issues_react_bots_timeline_pagination","issues_react_chrome_container_query_fix","issues_react_prohibit_title_fallback","issues_react_relay_cache_index","issues_react_timeline_side_panel","issues_search_type_gql","landing_pages_ninetailed","landing_pages_web_vitals_tracking","lifecycle_label_name_updates","low_quality_classifier","marketing_pages_search_explore_provider","memex_default_issue_create_repository","memex_live_update_hovercard","memex_mwl_filter_field_delimiter","memex_remove_deprecated_type_issue","merge_status_header_feedback","notifications_menu_defer_labels","oauth_authorize_clickjacking_protection","octocaptcha_origin_optimization","primer_react_overlay_max_height_clamp_to_viewport","primer_react_spinner_synchronize_animations","prs_conversations_react","rules_insights_filter_bar_created","sample_network_conn_type","secret_scanning_pattern_alerts_link","session_logs_ungroup_reasoning_text","site_features_copilot_universe","site_homepage_collaborate_video","spark_prompt_secret_scanning","spark_server_connection_status","suppress_automated_browser_vitals","viewscreen_sandbox","webp_support","workbench_store_readonly"],"copilotApiOverrideUrl":"https://api.githubcopilot.com"} GitHub - PacktPublishing/LLM-Engineers-Handbook: The LLM's practical guide: From the fundamentals to deploying advanced LLM and RAG apps to AWS using LLMOps best practices · GitHub Skip to content {"props":{"docsUrl":"https://docs.github.com/get-started/accessibility/keyboard-shortcuts"}} /* Override primer focus outline color for marketing header dropdown links for better contrast */ [data-color-mode="light"] .HeaderMenu-dropdown-link:focus-visible, [data-color-mode="light"] .HeaderMenu-trailing-link a:focus-visible { outline-color: var(--color-accent-fg); } Navigation Menu Toggle navigation Sign in Appearance settings {"props":{}} {"props":{"should_use_dotcom_links":true}} PlatformAI CODE CREATIONGitHub CopilotWrite better code with AIGitHub SparkBuild and deploy intelligent appsGitHub ModelsManage and compare promptsMCP RegistryNewIntegrate external toolsDEVELOPER WORKFLOWSActionsAutomate any workflowCodespacesInstant dev environmentsIssuesPlan and track workCode ReviewManage code changesAPPLICATION SECURITYGitHub Advanced SecurityFind and fix vulnerabilitiesCode securitySecure your code as you buildSecret protectionStop leaks before they startEXPLOREWhy GitHubDocumentationBlogChangelogMarketplaceView all featuresSolutionsBY COMPANY SIZEEnterprisesSmall and medium teamsStartupsNonprofitsBY USE CASEApp ModernizationDevSecOpsDevOpsCI/CDView all use casesBY INDUSTRYHealthcareFinancial servicesManufacturingGovernmentView all industriesView all solutionsResourcesEXPLORE BY TOPICAISoftware DevelopmentDevOpsSecurityView all topicsEXPLORE BY TYPECustomer storiesEvents & webinarsEbooks & reportsBusiness insightsGitHub SkillsSUPPORT & SERVICESDocumentationCustomer supportCommunity forumTrust centerPartnersView all resourcesOpen SourceCOMMUNITYGitHub SponsorsFund open source developersPROGRAMSSecurity LabMaintainer CommunityAcceleratorGitHub StarsArchive ProgramREPOSITORIESTopicsTrendingCollectionsEnterpriseENTERPRISE SOLUTIONSEnterprise platformAI-powered developer platformAVAILABLE ADD-ONSGitHub Advanced SecurityEnterprise-grade security featuresCopilot for BusinessEnterprise-grade AI featuresPremium SupportEnterprise-grade 24/7 supportPricing{"resolvedServerColorMode":"day"} Search or jump to... Search code, repositories, users, issues, pull requests... --> Search Clear Search syntax tips Provide feedback --> We read every piece of feedback, and take your input very seriously. Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly --> Name Query To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up Appearance settings {"props":{}} Resetting focus You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert {{ message }} PacktPublishing / LLM-Engineers-Handbook Public Notifications You must be signed in to change notification settings Fork 1.2k Star 4.9k Code Issues 23 Pull requests 8 Actions Security and quality 0 Insights Additional navigation options Code Issues Pull requests Actions Security and quality Insights {"payload":{"codeViewRepoRoute":{"path":"/","refInfo":{"name":"main","listCacheKey":"v0:1735803499.0","canEdit":false,"refType":"branch","currentOid":"e6038da300616020695f8effb3cbf0a78aac5870"},"tree":{"items":[{"name":".github/workflows","path":".github/workflows","contentType":"directory","hasSimplifiedPath":true},{"name":".vscode","path":".vscode","contentType":"directory"},{"name":"code_snippets","path":"code_snippets","contentType":"directory"},{"name":"configs","path":"configs","contentType":"directory"},{"name":"data","path":"data","contentType":"directory"},{"name":"images","path":"images","contentType":"directory"},{"name":"llm_engineering","path":"llm_engineering","contentType":"directory"},{"name":"pipelines","path":"pipelines","contentType":"directory"},{"name":"steps","path":"steps","contentType":"directory"},{"name":"tests","path":"tests","contentType":"directory"},{"name":"tools","path":"tools","contentType":"directory"},{"name":".env.example","path":".env.example","contentType":"file"},{"name":".gitignore","path":".gitignore","contentType":"file"},{"name":".pre-commit-config.yaml","path":".pre-commit-config.yaml","contentType":"file"},{"name":".python-version","path":".python-version","contentType":"file"},{"name":"Dockerfile","path":"Dockerfile","contentType":"file"},{"name":"LICENSE","path":"LICENSE","contentType":"file"},{"name":"README.md","path":"README.md","contentType":"file"},{"name":"docker-compose.yml","path":"docker-compose.yml","contentType":"file"},{"name":"poetry.lock","path":"poetry.lock","contentType":"file"},{"name":"pyproject.toml","path":"pyproject.toml","contentType":"file"},{"name":"ruff.toml","path":"ruff.toml","contentType":"file"}],"totalCount":22,"templateDirectorySuggestionUrl":null,"readme":null,"showBranchInfobar":false},"userNameDisplayConfiguration":"handle","treeExpanded":false,"symbolsExpanded":false,"copilotSWEAgentEnabled":false,"isOverview":true,"overview":{"banners":{"shouldRecommendReadme":false,"isPersonalRepo":false,"showUseActionBanner":false,"actionSlug":null,"actionId":null,"showProtectBranchBanner":false,"transactionalMessageBanner":null,"publishBannersInfo":{"dismissActionNoticePath":"/settings/dismiss-notice/publish_action_from_repo","releasePath":"/PacktPublishing/LLM-Engineers-Handbook/releases/new?marketplace=true","showPublishActionBanner":false},"interactionLimitBanner":null,"showInvitationBanner":false,"inviterName":null,"actionsMigrationBannerInfo":{"releaseTags":[],"showImmutableActionsMigrationBanner":false,"initialMigrationStatus":null}},"codeButton":{"contactPath":"/contact","isEnterprise":false,"local":{"protocolInfo":{"httpAvailable":true,"sshAvailable":null,"httpUrl":"https://github.com/PacktPublishing/LLM-Engineers-Handbook.git","showCloneWarning":null,"sshUrl":null,"sshCertificatesRequired":null,"sshCertificatesAvailable":null,"ghCliUrl":"gh repo clone PacktPublishing/LLM-Engineers-Handbook","defaultProtocol":"http","newSshKeyUrl":"/settings/ssh/new","setProtocolPath":"/users/set_protocol"},"platformInfo":{"cloneUrl":"https://desktop.github.com","showVisualStudioCloneButton":false,"visualStudioCloneUrl":"https://windows.github.com","showXcodeCloneButton":false,"xcodeCloneUrl":"xcode://clone?repo=https%3A%2F%2Fgithub.com%2FPacktPublishing%2FLLM-Engineers-Handbook","zipballUrl":"/PacktPublishing/LLM-Engineers-Handbook/archive/refs/heads/main.zip"}},"newCodespacePath":"/codespaces/new?hide_repo_select=true\u0026repo=784183059"},"popovers":{"rename":null,"renamedParentRepo":null},"commitCount":"201","overviewFiles":[{"displayName":"README.md","repoName":"LLM-Engineers-Handbook","refName":"main","path":"README.md","preferredFileType":"readme","tabName":"README","richText":"\u003carticle class=\"markdown-body entry-content container-lg\" itemprop=\"text\"\u003e\u003cp align=\"center\" dir=\"auto\"\u003e\u003ca href=\"https://www.eventbrite.com/e/ship-production-pytorch-system-in-a-day-train-optimize-deploy-workshop-tickets-1983348934052?aff=GitHub\" rel=\"nofollow\"\u003e\u003cimg src=\"https://camo.githubusercontent.com/b9c13fddfcb7d388660e7c60c3f7c67412deef8f3a83833a798b18b4b5efda38/68747470733a2f2f7374617469632e7061636b742d63646e2e636f6d2f6173736574732f696d616765732f7061636b742b6576656e74732f776f726b73686f705f6173686973685f6769746875622e706e67\" data-canonical-src=\"https://static.packt-cdn.com/assets/images/packt+events/workshop_ashish_github.png\" style=\"max-width: 100%;\"\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cp align=\"center\" dir=\"auto\"\u003e\u003ca href=\"https://www.packtpub.com/en-us/unlock?step=1\" rel=\"nofollow\"\u003e\u003cimg src=\"https://camo.githubusercontent.com/1d5684bdb5380c77317e1c2e8805f440f76c0771f523aa0c45a536cdfc10bf1e/68747470733a2f2f7374617469632e7061636b742d63646e2e636f6d2f6173736574732f696d616765732f7061636b742b6576656e74732f66696e616c47485f64657369676e5f72656465656d2e706e67\" data-canonical-src=\"https://static.packt-cdn.com/assets/images/packt+events/finalGH_design_redeem.png\" style=\"max-width: 100%;\"\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cdiv align=\"center\" dir=\"auto\"\u003e\n \u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch1 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003e👷 LLM Engineer's Handbook\u003c/h1\u003e\u003ca id=\"user-content--llm-engineers-handbook\" class=\"anchor\" aria-label=\"Permalink: 👷 LLM Engineer's Handbook\" href=\"#-llm-engineers-handbook\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n \u003cp dir=\"auto\"\u003eOfficial repository of the \u003ca href=\"https://www.amazon.com/LLM-Engineers-Handbook-engineering-production/dp/1836200072/\" rel=\"nofollow\"\u003eLLM Engineer's Handbook\u003c/a\u003e by \u003ca href=\"https://github.com/iusztinpaul\"\u003ePaul Iusztin\u003c/a\u003e and \u003ca href=\"https://github.com/mlabonne\"\u003eMaxime Labonne\u003c/a\u003e\u003c/p\u003e\n\u003c/div\u003e\n\u003cbr\u003e\n\u003cp align=\"center\" dir=\"auto\"\u003e\n \u003ca href=\"https://www.amazon.com/LLM-Engineers-Handbook-engineering-production/dp/1836200072/\" rel=\"nofollow\"\u003e\n \u003cimg src=\"/PacktPublishing/LLM-Engineers-Handbook/raw/main/images/cover_plus.png\" alt=\"Book cover\" style=\"max-width: 100%;\"\u003e\n \u003c/a\u003e\n\u003c/p\u003e\n\u003cp align=\"center\" dir=\"auto\"\u003e\n Find the book on \u003ca href=\"https://www.amazon.com/LLM-Engineers-Handbook-engineering-production/dp/1836200072/\" rel=\"nofollow\"\u003eAmazon\u003c/a\u003e or \u003ca href=\"https://www.packtpub.com/en-us/product/llm-engineers-handbook-9781836200062\" rel=\"nofollow\"\u003ePackt\u003c/a\u003e\n\u003c/p\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch2 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003e🌟 Features\u003c/h2\u003e\u003ca id=\"user-content--features\" class=\"anchor\" aria-label=\"Permalink: 🌟 Features\" href=\"#-features\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eThe goal of this book is to create your own end-to-end LLM-based system using best practices:\u003c/p\u003e\n\u003cul dir=\"auto\"\u003e\n\u003cli\u003e📝 Data collection \u0026amp; generation\u003c/li\u003e\n\u003cli\u003e🔄 LLM training pipeline\u003c/li\u003e\n\u003cli\u003e📊 Simple RAG system\u003c/li\u003e\n\u003cli\u003e🚀 Production-ready AWS deployment\u003c/li\u003e\n\u003cli\u003e🔍 Comprehensive monitoring\u003c/li\u003e\n\u003cli\u003e🧪 Testing and evaluation framework\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp dir=\"auto\"\u003eYou can download and use the final trained model on \u003ca href=\"https://huggingface.co/mlabonne/TwinLlama-3.1-8B-DPO\" rel=\"nofollow\"\u003eHugging Face\u003c/a\u003e.\u003c/p\u003e\n\u003cdiv class=\"markdown-alert markdown-alert-important\" dir=\"auto\"\u003e\u003cp class=\"markdown-alert-title\" dir=\"auto\"\u003e\u003csvg class=\"octicon octicon-report mr-2\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"M0 1.75C0 .784.784 0 1.75 0h12.5C15.216 0 16 .784 16 1.75v9.5A1.75 1.75 0 0 1 14.25 13H8.06l-2.573 2.573A1.458 1.458 0 0 1 3 14.543V13H1.75A1.75 1.75 0 0 1 0 11.25Zm1.75-.25a.25.25 0 0 0-.25.25v9.5c0 .138.112.25.25.25h2a.75.75 0 0 1 .75.75v2.19l2.72-2.72a.749.749 0 0 1 .53-.22h6.5a.25.25 0 0 0 .25-.25v-9.5a.25.25 0 0 0-.25-.25Zm7 2.25v2.5a.75.75 0 0 1-1.5 0v-2.5a.75.75 0 0 1 1.5 0ZM9 9a1 1 0 1 1-2 0 1 1 0 0 1 2 0Z\"\u003e\u003c/path\u003e\u003c/svg\u003eImportant\u003c/p\u003e\u003cp dir=\"auto\"\u003eThe code in this GitHub repository is actively maintained and may contain updates not reflected in the book. \u003cstrong\u003eAlways refer to this repository for the latest version of the code.\u003c/strong\u003e\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch2 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003e🔗 Dependencies\u003c/h2\u003e\u003ca id=\"user-content--dependencies\" class=\"anchor\" aria-label=\"Permalink: 🔗 Dependencies\" href=\"#-dependencies\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch3 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eLocal dependencies\u003c/h3\u003e\u003ca id=\"user-content-local-dependencies\" class=\"anchor\" aria-label=\"Permalink: Local dependencies\" href=\"#local-dependencies\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eTo install and run the project locally, you need the following dependencies.\u003c/p\u003e\n\u003cmarkdown-accessiblity-table\u003e\u003ctable\u003e\n\u003cthead\u003e\n\u003ctr\u003e\n\u003cth\u003eTool\u003c/th\u003e\n\u003cth\u003eVersion\u003c/th\u003e\n\u003cth\u003ePurpose\u003c/th\u003e\n\u003cth\u003eInstallation Link\u003c/th\u003e\n\u003c/tr\u003e\n\u003c/thead\u003e\n\u003ctbody\u003e\n\u003ctr\u003e\n\u003ctd\u003epyenv\u003c/td\u003e\n\u003ctd\u003e≥2.3.36\u003c/td\u003e\n\u003ctd\u003eMultiple Python versions (optional)\u003c/td\u003e\n\u003ctd\u003e\u003ca href=\"https://github.com/pyenv/pyenv?tab=readme-ov-file#installation\"\u003eInstall Guide\u003c/a\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003ePython\u003c/td\u003e\n\u003ctd\u003e3.11\u003c/td\u003e\n\u003ctd\u003eRuntime environment\u003c/td\u003e\n\u003ctd\u003e\u003ca href=\"https://www.python.org/downloads/\" rel=\"nofollow\"\u003eDownload\u003c/a\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003ePoetry\u003c/td\u003e\n\u003ctd\u003e\u0026gt;= 1.8.3 and \u0026lt; 2.0\u003c/td\u003e\n\u003ctd\u003ePackage management\u003c/td\u003e\n\u003ctd\u003e\u003ca href=\"https://python-poetry.org/docs/#installation\" rel=\"nofollow\"\u003eInstall Guide\u003c/a\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003eDocker\u003c/td\u003e\n\u003ctd\u003e≥27.1.1\u003c/td\u003e\n\u003ctd\u003eContainerization\u003c/td\u003e\n\u003ctd\u003e\u003ca href=\"https://docs.docker.com/engine/install/\" rel=\"nofollow\"\u003eInstall Guide\u003c/a\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003eAWS CLI\u003c/td\u003e\n\u003ctd\u003e≥2.15.42\u003c/td\u003e\n\u003ctd\u003eCloud management\u003c/td\u003e\n\u003ctd\u003e\u003ca href=\"https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html\" rel=\"nofollow\"\u003eInstall Guide\u003c/a\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003eGit\u003c/td\u003e\n\u003ctd\u003e≥2.44.0\u003c/td\u003e\n\u003ctd\u003eVersion control\u003c/td\u003e\n\u003ctd\u003e\u003ca href=\"https://git-scm.com/downloads\" rel=\"nofollow\"\u003eDownload\u003c/a\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/tbody\u003e\n\u003c/table\u003e\u003c/markdown-accessiblity-table\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch3 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eCloud services\u003c/h3\u003e\u003ca id=\"user-content-cloud-services\" class=\"anchor\" aria-label=\"Permalink: Cloud services\" href=\"#cloud-services\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eThe code also uses and depends on the following cloud services. For now, you don't have to do anything. We will guide you in the installation and deployment sections on how to use them:\u003c/p\u003e\n\u003cmarkdown-accessiblity-table\u003e\u003ctable\u003e\n\u003cthead\u003e\n\u003ctr\u003e\n\u003cth\u003eService\u003c/th\u003e\n\u003cth\u003ePurpose\u003c/th\u003e\n\u003c/tr\u003e\n\u003c/thead\u003e\n\u003ctbody\u003e\n\u003ctr\u003e\n\u003ctd\u003e\u003ca href=\"https://huggingface.com/\" rel=\"nofollow\"\u003eHuggingFace\u003c/a\u003e\u003c/td\u003e\n\u003ctd\u003eModel registry\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\u003ca href=\"https://www.comet.com/site/products/opik/?utm_source=llm_handbook\u0026amp;utm_medium=github\u0026amp;utm_campaign=opik\" rel=\"nofollow\"\u003eComet ML\u003c/a\u003e\u003c/td\u003e\n\u003ctd\u003eExperiment tracker\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\u003ca href=\"https://www.comet.com/site/products/opik/?utm_source=llm_handbook\u0026amp;utm_medium=github\u0026amp;utm_campaign=opik\" rel=\"nofollow\"\u003eOpik\u003c/a\u003e\u003c/td\u003e\n\u003ctd\u003ePrompt monitoring\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\u003ca href=\"https://www.zenml.io/\" rel=\"nofollow\"\u003eZenML\u003c/a\u003e\u003c/td\u003e\n\u003ctd\u003eOrchestrator and artifacts layer\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\u003ca href=\"https://aws.amazon.com/\" rel=\"nofollow\"\u003eAWS\u003c/a\u003e\u003c/td\u003e\n\u003ctd\u003eCompute and storage\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\u003ca href=\"https://www.mongodb.com/\" rel=\"nofollow\"\u003eMongoDB\u003c/a\u003e\u003c/td\u003e\n\u003ctd\u003eNoSQL database\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\u003ca href=\"https://qdrant.tech/\" rel=\"nofollow\"\u003eQdrant\u003c/a\u003e\u003c/td\u003e\n\u003ctd\u003eVector database\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\u003ca href=\"https://github.com/features/actions\"\u003eGitHub Actions\u003c/a\u003e\u003c/td\u003e\n\u003ctd\u003eCI/CD pipeline\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/tbody\u003e\n\u003c/table\u003e\u003c/markdown-accessiblity-table\u003e\n\u003cp dir=\"auto\"\u003eIn the \u003ca href=\"https://www.amazon.com/LLM-Engineers-Handbook-engineering-production/dp/1836200072/\" rel=\"nofollow\"\u003eLLM Engineer's Handbook\u003c/a\u003e, Chapter 2 will walk you through each tool. Chapters 10 and 11 provide step-by-step guides on how to set up everything you need.\u003c/p\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch2 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003e🗂️ Project Structure\u003c/h2\u003e\u003ca id=\"user-content-️-project-structure\" class=\"anchor\" aria-label=\"Permalink: 🗂️ Project Structure\" href=\"#️-project-structure\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eHere is the directory overview:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\".\n├── code_snippets/ # Standalone example code\n├── configs/ # Pipeline configuration files\n├── llm_engineering/ # Core project package\n│ ├── application/ \n│ ├── domain/ \n│ ├── infrastructure/ \n│ ├── model/ \n├── pipelines/ # ML pipeline definitions\n├── steps/ # Pipeline components\n├── tests/ # Test examples\n├── tools/ # Utility scripts\n│ ├── run.py\n│ ├── ml_service.py\n│ ├── rag.py\n│ ├── data_warehouse.py\"\u003e\u003cpre\u003e\u003cspan class=\"pl-c1\"\u003e.\u003c/span\u003e\n├── code_snippets/ \u003cspan class=\"pl-c\"\u003e\u003cspan class=\"pl-c\"\u003e#\u003c/span\u003e Standalone example code\u003c/span\u003e\n├── configs/ \u003cspan class=\"pl-c\"\u003e\u003cspan class=\"pl-c\"\u003e#\u003c/span\u003e Pipeline configuration files\u003c/span\u003e\n├── llm_engineering/ \u003cspan class=\"pl-c\"\u003e\u003cspan class=\"pl-c\"\u003e#\u003c/span\u003e Core project package\u003c/span\u003e\n│ ├── application/ \n│ ├── domain/ \n│ ├── infrastructure/ \n│ ├── model/ \n├── pipelines/ \u003cspan class=\"pl-c\"\u003e\u003cspan class=\"pl-c\"\u003e#\u003c/span\u003e ML pipeline definitions\u003c/span\u003e\n├── steps/ \u003cspan class=\"pl-c\"\u003e\u003cspan class=\"pl-c\"\u003e#\u003c/span\u003e Pipeline components\u003c/span\u003e\n├── tests/ \u003cspan class=\"pl-c\"\u003e\u003cspan class=\"pl-c\"\u003e#\u003c/span\u003e Test examples\u003c/span\u003e\n├── tools/ \u003cspan class=\"pl-c\"\u003e\u003cspan class=\"pl-c\"\u003e#\u003c/span\u003e Utility scripts\u003c/span\u003e\n│ ├── run.py\n│ ├── ml_service.py\n│ ├── rag.py\n│ ├── data_warehouse.py\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003e\u003ccode\u003ellm_engineering/\u003c/code\u003e is the main Python package implementing LLM and RAG functionality. It follows Domain-Driven Design (DDD) principles:\u003c/p\u003e\n\u003cul dir=\"auto\"\u003e\n\u003cli\u003e\u003ccode\u003edomain/\u003c/code\u003e: Core business entities and structures\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eapplication/\u003c/code\u003e: Business logic, crawlers, and RAG implementation\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003emodel/\u003c/code\u003e: LLM training and inference\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003einfrastructure/\u003c/code\u003e: External service integrations (AWS, Qdrant, MongoDB, FastAPI)\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp dir=\"auto\"\u003eThe code logic and imports flow as follows: \u003ccode\u003einfrastructure\u003c/code\u003e → \u003ccode\u003emodel\u003c/code\u003e → \u003ccode\u003eapplication\u003c/code\u003e → \u003ccode\u003edomain\u003c/code\u003e\u003c/p\u003e\n\u003cp dir=\"auto\"\u003e\u003ccode\u003epipelines/\u003c/code\u003e: Contains the ZenML ML pipelines, which serve as the entry point for all the ML pipelines. Coordinates the data processing and model training stages of the ML lifecycle.\u003c/p\u003e\n\u003cp dir=\"auto\"\u003e\u003ccode\u003esteps/\u003c/code\u003e: Contains individual ZenML steps, which are reusable components for building and customizing ZenML pipelines. Steps perform specific tasks (e.g., data loading, preprocessing) and can be combined within the ML pipelines.\u003c/p\u003e\n\u003cp dir=\"auto\"\u003e\u003ccode\u003etests/\u003c/code\u003e: Covers a few sample tests used as examples within the CI pipeline.\u003c/p\u003e\n\u003cp dir=\"auto\"\u003e\u003ccode\u003etools/\u003c/code\u003e: Utility scripts used to call the ZenML pipelines and inference code:\u003c/p\u003e\n\u003cul dir=\"auto\"\u003e\n\u003cli\u003e\u003ccode\u003erun.py\u003c/code\u003e: Entry point script to run ZenML pipelines.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eml_service.py\u003c/code\u003e: Starts the REST API inference server.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003erag.py\u003c/code\u003e: Demonstrates usage of the RAG retrieval module.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003edata_warehouse.py\u003c/code\u003e: Used to export or import data from the MongoDB data warehouse through JSON files.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp dir=\"auto\"\u003e\u003ccode\u003econfigs/\u003c/code\u003e: ZenML YAML configuration files to control the execution of pipelines and steps.\u003c/p\u003e\n\u003cp dir=\"auto\"\u003e\u003ccode\u003ecode_snippets/\u003c/code\u003e: Independent code examples that can be executed independently.\u003c/p\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch2 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003e💻 Installation\u003c/h2\u003e\u003ca id=\"user-content--installation\" class=\"anchor\" aria-label=\"Permalink: 💻 Installation\" href=\"#-installation\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cdiv class=\"markdown-alert markdown-alert-note\" dir=\"auto\"\u003e\u003cp class=\"markdown-alert-title\" dir=\"auto\"\u003e\u003csvg class=\"octicon octicon-info mr-2\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"M0 8a8 8 0 1 1 16 0A8 8 0 0 1 0 8Zm8-6.5a6.5 6.5 0 1 0 0 13 6.5 6.5 0 0 0 0-13ZM6.5 7.75A.75.75 0 0 1 7.25 7h1a.75.75 0 0 1 .75.75v2.75h.25a.75.75 0 0 1 0 1.5h-2a.75.75 0 0 1 0-1.5h.25v-2h-.25a.75.75 0 0 1-.75-.75ZM8 6a1 1 0 1 1 0-2 1 1 0 0 1 0 2Z\"\u003e\u003c/path\u003e\u003c/svg\u003eNote\u003c/p\u003e\u003cp dir=\"auto\"\u003eIf you are experiencing issues while installing and running the repository, consider checking the \u003ca href=\"https://github.com/PacktPublishing/LLM-Engineers-Handbook/issues\"\u003eIssues\u003c/a\u003e GitHub section for other people who solved similar problems or directly asking us for help.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch3 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003e1. Clone the Repository\u003c/h3\u003e\u003ca id=\"user-content-1-clone-the-repository\" class=\"anchor\" aria-label=\"Permalink: 1. Clone the Repository\" href=\"#1-clone-the-repository\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eStart by cloning the repository and navigating to the project directory:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"git clone https://github.com/PacktPublishing/LLM-Engineers-Handbook.git\ncd LLM-Engineers-Handbook \"\u003e\u003cpre\u003egit clone https://github.com/PacktPublishing/LLM-Engineers-Handbook.git\n\u003cspan class=\"pl-c1\"\u003ecd\u003c/span\u003e LLM-Engineers-Handbook \u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eNext, we have to prepare your Python environment and its adjacent dependencies.\u003c/p\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch3 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003e2. Set Up Python Environment\u003c/h3\u003e\u003ca id=\"user-content-2-set-up-python-environment\" class=\"anchor\" aria-label=\"Permalink: 2. Set Up Python Environment\" href=\"#2-set-up-python-environment\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eThe project requires Python 3.11. You can either use your global Python installation or set up a project-specific version using pyenv.\u003c/p\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch4 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eOption A: Using Global Python (if version 3.11 is installed)\u003c/h4\u003e\u003ca id=\"user-content-option-a-using-global-python-if-version-311-is-installed\" class=\"anchor\" aria-label=\"Permalink: Option A: Using Global Python (if version 3.11 is installed)\" href=\"#option-a-using-global-python-if-version-311-is-installed\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eVerify your Python version:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"python --version # Should show Python 3.11.x\"\u003e\u003cpre\u003epython --version \u003cspan class=\"pl-c\"\u003e\u003cspan class=\"pl-c\"\u003e#\u003c/span\u003e Should show Python 3.11.x\u003c/span\u003e\u003c/pre\u003e\u003c/div\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch4 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eOption B: Using pyenv (recommended)\u003c/h4\u003e\u003ca id=\"user-content-option-b-using-pyenv-recommended\" class=\"anchor\" aria-label=\"Permalink: Option B: Using pyenv (recommended)\" href=\"#option-b-using-pyenv-recommended\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003col dir=\"auto\"\u003e\n\u003cli\u003eVerify pyenv installation:\u003c/li\u003e\n\u003c/ol\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"pyenv --version # Should show pyenv 2.3.36 or later\"\u003e\u003cpre\u003epyenv --version \u003cspan class=\"pl-c\"\u003e\u003cspan class=\"pl-c\"\u003e#\u003c/span\u003e Should show pyenv 2.3.36 or later\u003c/span\u003e\u003c/pre\u003e\u003c/div\u003e\n\u003col start=\"2\" dir=\"auto\"\u003e\n\u003cli\u003eInstall Python 3.11.8:\u003c/li\u003e\n\u003c/ol\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"pyenv install 3.11.8\"\u003e\u003cpre\u003epyenv install 3.11.8\u003c/pre\u003e\u003c/div\u003e\n\u003col start=\"3\" dir=\"auto\"\u003e\n\u003cli\u003eVerify the installation:\u003c/li\u003e\n\u003c/ol\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"python --version # Should show Python 3.11.8\"\u003e\u003cpre\u003epython --version \u003cspan class=\"pl-c\"\u003e\u003cspan class=\"pl-c\"\u003e#\u003c/span\u003e Should show Python 3.11.8\u003c/span\u003e\u003c/pre\u003e\u003c/div\u003e\n\u003col start=\"4\" dir=\"auto\"\u003e\n\u003cli\u003eConfirm Python version in the project directory:\u003c/li\u003e\n\u003c/ol\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"python --version\n# Output: Python 3.11.8\"\u003e\u003cpre\u003epython --version\n\u003cspan class=\"pl-c\"\u003e\u003cspan class=\"pl-c\"\u003e#\u003c/span\u003e Output: Python 3.11.8\u003c/span\u003e\u003c/pre\u003e\u003c/div\u003e\n\u003cdiv class=\"markdown-alert markdown-alert-note\" dir=\"auto\"\u003e\u003cp class=\"markdown-alert-title\" dir=\"auto\"\u003e\u003csvg class=\"octicon octicon-info mr-2\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"M0 8a8 8 0 1 1 16 0A8 8 0 0 1 0 8Zm8-6.5a6.5 6.5 0 1 0 0 13 6.5 6.5 0 0 0 0-13ZM6.5 7.75A.75.75 0 0 1 7.25 7h1a.75.75 0 0 1 .75.75v2.75h.25a.75.75 0 0 1 0 1.5h-2a.75.75 0 0 1 0-1.5h.25v-2h-.25a.75.75 0 0 1-.75-.75ZM8 6a1 1 0 1 1 0-2 1 1 0 0 1 0 2Z\"\u003e\u003c/path\u003e\u003c/svg\u003eNote\u003c/p\u003e\u003cp dir=\"auto\"\u003eThe project includes a \u003ccode\u003e.python-version\u003c/code\u003e file that automatically sets the correct Python version when you're in the project directory.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch3 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003e3. Install Dependencies\u003c/h3\u003e\u003ca id=\"user-content-3-install-dependencies\" class=\"anchor\" aria-label=\"Permalink: 3. Install Dependencies\" href=\"#3-install-dependencies\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eThe project uses Poetry for dependency management.\u003c/p\u003e\n\u003col dir=\"auto\"\u003e\n\u003cli\u003eVerify Poetry installation:\u003c/li\u003e\n\u003c/ol\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry --version # Should show Poetry version 1.8.3 or later\"\u003e\u003cpre\u003epoetry --version \u003cspan class=\"pl-c\"\u003e\u003cspan class=\"pl-c\"\u003e#\u003c/span\u003e Should show Poetry version 1.8.3 or later\u003c/span\u003e\u003c/pre\u003e\u003c/div\u003e\n\u003col start=\"2\" dir=\"auto\"\u003e\n\u003cli\u003eSet up the project environment and install dependencies:\u003c/li\u003e\n\u003c/ol\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry env use 3.11\npoetry install --without aws\npoetry run pre-commit install\"\u003e\u003cpre\u003epoetry env use 3.11\npoetry install --without aws\npoetry run pre-commit install\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eThis will:\u003c/p\u003e\n\u003cul dir=\"auto\"\u003e\n\u003cli\u003eConfigure Poetry to use Python 3.11\u003c/li\u003e\n\u003cli\u003eInstall project dependencies (excluding AWS-specific packages)\u003c/li\u003e\n\u003cli\u003eSet up pre-commit hooks for code verification\u003c/li\u003e\n\u003c/ul\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch3 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003e4. Activate the Environment\u003c/h3\u003e\u003ca id=\"user-content-4-activate-the-environment\" class=\"anchor\" aria-label=\"Permalink: 4. Activate the Environment\" href=\"#4-activate-the-environment\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eAs our task manager, we run all the scripts using \u003ca href=\"https://poethepoet.natn.io/index.html\" rel=\"nofollow\"\u003ePoe the Poet\u003c/a\u003e.\u003c/p\u003e\n\u003col dir=\"auto\"\u003e\n\u003cli\u003eStart a Poetry shell:\u003c/li\u003e\n\u003c/ol\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry shell\"\u003e\u003cpre\u003epoetry shell\u003c/pre\u003e\u003c/div\u003e\n\u003col start=\"2\" dir=\"auto\"\u003e\n\u003cli\u003eRun project commands using Poe the Poet:\u003c/li\u003e\n\u003c/ol\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe ...\"\u003e\u003cpre\u003epoetry poe ...\u003c/pre\u003e\u003c/div\u003e\n\u003cdetails\u003e\n\u003csummary\u003e🔧 Troubleshooting Poe the Poet Installation\u003c/summary\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch3 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eAlternative Command Execution\u003c/h3\u003e\u003ca id=\"user-content-alternative-command-execution\" class=\"anchor\" aria-label=\"Permalink: Alternative Command Execution\" href=\"#alternative-command-execution\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eIf you're experiencing issues with \u003ccode\u003epoethepoet\u003c/code\u003e, you can still run the project commands directly through Poetry. Here's how:\u003c/p\u003e\n\u003col dir=\"auto\"\u003e\n\u003cli\u003eLook up the command definition in \u003ccode\u003epyproject.toml\u003c/code\u003e\u003c/li\u003e\n\u003cli\u003eUse \u003ccode\u003epoetry run\u003c/code\u003e with the underlying command\u003c/li\u003e\n\u003c/ol\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch4 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eExample:\u003c/h4\u003e\u003ca id=\"user-content-example\" class=\"anchor\" aria-label=\"Permalink: Example:\" href=\"#example\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eInstead of:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe local-infrastructure-up\"\u003e\u003cpre\u003epoetry poe local-infrastructure-up\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eUse the direct command from pyproject.toml:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry run \u0026lt;actual-command-from-pyproject-toml\u0026gt;\"\u003e\u003cpre\u003epoetry run \u003cspan class=\"pl-k\"\u003e\u0026lt;\u003c/span\u003eactual-command-from-pyproject-toml\u003cspan class=\"pl-k\"\u003e\u0026gt;\u003c/span\u003e\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eNote: All project commands are defined in the [tool.poe.tasks] section of pyproject.toml\u003c/p\u003e\n\u003c/details\u003e\n\u003cp dir=\"auto\"\u003eNow, let's configure our local project with all the necessary credentials and tokens to run the code locally.\u003c/p\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch3 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003e5. Local Development Setup\u003c/h3\u003e\u003ca id=\"user-content-5-local-development-setup\" class=\"anchor\" aria-label=\"Permalink: 5. Local Development Setup\" href=\"#5-local-development-setup\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eAfter you have installed all the dependencies, you must create and fill a \u003ccode\u003e.env\u003c/code\u003e file with your credentials to appropriately interact with other services and run the project. Setting your sensitive credentials in a \u003ccode\u003e.env\u003c/code\u003e file is a good security practice, as this file won't be committed to GitHub or shared with anyone else.\u003c/p\u003e\n\u003col dir=\"auto\"\u003e\n\u003cli\u003eFirst, copy our example by running the following:\u003c/li\u003e\n\u003c/ol\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"cp .env.example .env # The file must be at your repository's root!\"\u003e\u003cpre\u003ecp .env.example .env \u003cspan class=\"pl-c\"\u003e\u003cspan class=\"pl-c\"\u003e#\u003c/span\u003e The file must be at your repository's root!\u003c/span\u003e\u003c/pre\u003e\u003c/div\u003e\n\u003col start=\"2\" dir=\"auto\"\u003e\n\u003cli\u003eNow, let's understand how to fill in all the essential variables within the \u003ccode\u003e.env\u003c/code\u003e file to get you started. The following are the mandatory settings we must complete when working locally:\u003c/li\u003e\n\u003c/ol\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch4 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eOpenAI\u003c/h4\u003e\u003ca id=\"user-content-openai\" class=\"anchor\" aria-label=\"Permalink: OpenAI\" href=\"#openai\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eTo authenticate to OpenAI's API, you must fill out the \u003ccode\u003eOPENAI_API_KEY\u003c/code\u003e env var with an authentication token.\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-dotenv notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"OPENAI_API_KEY=your_api_key_here\"\u003e\u003cpre\u003e\u003cspan class=\"pl-v\"\u003eOPENAI_API_KEY\u003c/span\u003e\u003cspan class=\"pl-k\"\u003e=\u003c/span\u003e\u003cspan class=\"pl-s\"\u003eyour_api_key_here\u003c/span\u003e\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003e→ Check out this \u003ca href=\"https://platform.openai.com/docs/quickstart\" rel=\"nofollow\"\u003etutorial\u003c/a\u003e to learn how to provide one from OpenAI.\u003c/p\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch4 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eHugging Face\u003c/h4\u003e\u003ca id=\"user-content-hugging-face\" class=\"anchor\" aria-label=\"Permalink: Hugging Face\" href=\"#hugging-face\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eTo authenticate to Hugging Face, you must fill out the \u003ccode\u003eHUGGINGFACE_ACCESS_TOKEN\u003c/code\u003e env var with an authentication token.\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-dotenv notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"HUGGINGFACE_ACCESS_TOKEN=your_token_here\"\u003e\u003cpre\u003e\u003cspan class=\"pl-v\"\u003eHUGGINGFACE_ACCESS_TOKEN\u003c/span\u003e\u003cspan class=\"pl-k\"\u003e=\u003c/span\u003e\u003cspan class=\"pl-s\"\u003eyour_token_here\u003c/span\u003e\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003e→ Check out this \u003ca href=\"https://huggingface.co/docs/hub/en/security-tokens\" rel=\"nofollow\"\u003etutorial\u003c/a\u003e to learn how to provide one from Hugging Face.\u003c/p\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch4 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eComet ML \u0026amp; Opik\u003c/h4\u003e\u003ca id=\"user-content-comet-ml--opik\" class=\"anchor\" aria-label=\"Permalink: Comet ML \u0026amp; Opik\" href=\"#comet-ml--opik\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eTo authenticate to Comet ML (required only during training) and Opik, you must fill out the \u003ccode\u003eCOMET_API_KEY\u003c/code\u003e env var with your authentication token.\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-dotenv notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"COMET_API_KEY=your_api_key_here\"\u003e\u003cpre\u003e\u003cspan class=\"pl-v\"\u003eCOMET_API_KEY\u003c/span\u003e\u003cspan class=\"pl-k\"\u003e=\u003c/span\u003e\u003cspan class=\"pl-s\"\u003eyour_api_key_here\u003c/span\u003e\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003e→ Check out this \u003ca href=\"https://www.comet.com/docs/opik/?utm_source=llm_handbook\u0026amp;utm_medium=github\u0026amp;utm_campaign=opik\" rel=\"nofollow\"\u003etutorial\u003c/a\u003e to learn how to get started with Opik. You can also access Opik's dashboard using 🔗\u003ca href=\"https://www.comet.com/opik?utm_source=llm_handbook\u0026amp;utm_medium=github\u0026amp;utm_content=opik\" rel=\"nofollow\"\u003ethis link\u003c/a\u003e.\u003c/p\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch3 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003e6. Deployment Setup\u003c/h3\u003e\u003ca id=\"user-content-6-deployment-setup\" class=\"anchor\" aria-label=\"Permalink: 6. Deployment Setup\" href=\"#6-deployment-setup\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eWhen deploying the project to the cloud, we must set additional settings for Mongo, Qdrant, and AWS. If you are just working locally, the default values of these env vars will work out of the box. Detailed deployment instructions are available in Chapter 11 of the \u003ca href=\"https://www.amazon.com/LLM-Engineers-Handbook-engineering-production/dp/1836200072/\" rel=\"nofollow\"\u003eLLM Engineer's Handbook\u003c/a\u003e.\u003c/p\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch4 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eMongoDB\u003c/h4\u003e\u003ca id=\"user-content-mongodb\" class=\"anchor\" aria-label=\"Permalink: MongoDB\" href=\"#mongodb\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eWe must change the \u003ccode\u003eDATABASE_HOST\u003c/code\u003e env var with the URL pointing to your cloud MongoDB cluster.\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-dotenv notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"DATABASE_HOST=your_mongodb_url\"\u003e\u003cpre\u003e\u003cspan class=\"pl-v\"\u003eDATABASE_HOST\u003c/span\u003e\u003cspan class=\"pl-k\"\u003e=\u003c/span\u003e\u003cspan class=\"pl-s\"\u003eyour_mongodb_url\u003c/span\u003e\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003e→ Check out this \u003ca href=\"https://www.mongodb.com/resources/products/fundamentals/mongodb-cluster-setup\" rel=\"nofollow\"\u003etutorial\u003c/a\u003e to learn how to create and host a MongoDB cluster for free.\u003c/p\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch4 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eQdrant\u003c/h4\u003e\u003ca id=\"user-content-qdrant\" class=\"anchor\" aria-label=\"Permalink: Qdrant\" href=\"#qdrant\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eChange \u003ccode\u003eUSE_QDRANT_CLOUD\u003c/code\u003e to \u003ccode\u003etrue\u003c/code\u003e, \u003ccode\u003eQDRANT_CLOUD_URL\u003c/code\u003e with the URL point to your cloud Qdrant cluster, and \u003ccode\u003eQDRANT_APIKEY\u003c/code\u003e with its API key.\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-dotenv notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"USE_QDRANT_CLOUD=true\nQDRANT_CLOUD_URL=your_qdrant_cloud_url\nQDRANT_APIKEY=your_qdrant_api_key\"\u003e\u003cpre\u003e\u003cspan class=\"pl-v\"\u003eUSE_QDRANT_CLOUD\u003c/span\u003e\u003cspan class=\"pl-k\"\u003e=\u003c/span\u003e\u003cspan class=\"pl-s\"\u003etrue\u003c/span\u003e\n\u003cspan class=\"pl-v\"\u003eQDRANT_CLOUD_URL\u003c/span\u003e\u003cspan class=\"pl-k\"\u003e=\u003c/span\u003e\u003cspan class=\"pl-s\"\u003eyour_qdrant_cloud_url\u003c/span\u003e\n\u003cspan class=\"pl-v\"\u003eQDRANT_APIKEY\u003c/span\u003e\u003cspan class=\"pl-k\"\u003e=\u003c/span\u003e\u003cspan class=\"pl-s\"\u003eyour_qdrant_api_key\u003c/span\u003e\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003e→ Check out this \u003ca href=\"https://qdrant.tech/documentation/cloud/create-cluster/\" rel=\"nofollow\"\u003etutorial\u003c/a\u003e to learn how to create a Qdrant cluster for free\u003c/p\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch4 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eAWS\u003c/h4\u003e\u003ca id=\"user-content-aws\" class=\"anchor\" aria-label=\"Permalink: AWS\" href=\"#aws\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eFor your AWS set-up to work correctly, you need the AWS CLI installed on your local machine and properly configured with an admin user (or a user with enough permissions to create new SageMaker, ECR, and S3 resources; using an admin user will make everything more straightforward).\u003c/p\u003e\n\u003cp dir=\"auto\"\u003eChapter 2 provides step-by-step instructions on how to install the AWS CLI, create an admin user on AWS, and get an access key to set up the \u003ccode\u003eAWS_ACCESS_KEY\u003c/code\u003e and \u003ccode\u003eAWS_SECRET_KEY\u003c/code\u003e environment variables. If you already have an AWS admin user in place, you have to configure the following env vars in your \u003ccode\u003e.env\u003c/code\u003e file:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"AWS_REGION=eu-central-1 # Change it with your AWS region.\nAWS_ACCESS_KEY=your_aws_access_key\nAWS_SECRET_KEY=your_aws_secret_key\"\u003e\u003cpre\u003eAWS_REGION=eu-central-1 \u003cspan class=\"pl-c\"\u003e\u003cspan class=\"pl-c\"\u003e#\u003c/span\u003e Change it with your AWS region.\u003c/span\u003e\nAWS_ACCESS_KEY=your_aws_access_key\nAWS_SECRET_KEY=your_aws_secret_key\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eAWS credentials are typically stored in \u003ccode\u003e~/.aws/credentials\u003c/code\u003e. You can view this file directly using \u003ccode\u003ecat\u003c/code\u003e or similar commands:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"cat ~/.aws/credentials\"\u003e\u003cpre\u003ecat \u003cspan class=\"pl-k\"\u003e~\u003c/span\u003e/.aws/credentials\u003c/pre\u003e\u003c/div\u003e\n\u003cdiv class=\"markdown-alert markdown-alert-important\" dir=\"auto\"\u003e\u003cp class=\"markdown-alert-title\" dir=\"auto\"\u003e\u003csvg class=\"octicon octicon-report mr-2\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"M0 1.75C0 .784.784 0 1.75 0h12.5C15.216 0 16 .784 16 1.75v9.5A1.75 1.75 0 0 1 14.25 13H8.06l-2.573 2.573A1.458 1.458 0 0 1 3 14.543V13H1.75A1.75 1.75 0 0 1 0 11.25Zm1.75-.25a.25.25 0 0 0-.25.25v9.5c0 .138.112.25.25.25h2a.75.75 0 0 1 .75.75v2.19l2.72-2.72a.749.749 0 0 1 .53-.22h6.5a.25.25 0 0 0 .25-.25v-9.5a.25.25 0 0 0-.25-.25Zm7 2.25v2.5a.75.75 0 0 1-1.5 0v-2.5a.75.75 0 0 1 1.5 0ZM9 9a1 1 0 1 1-2 0 1 1 0 0 1 2 0Z\"\u003e\u003c/path\u003e\u003c/svg\u003eImportant\u003c/p\u003e\u003cp dir=\"auto\"\u003eAdditional configuration options are available in \u003ca href=\"https://github.com/PacktPublishing/LLM-Engineers-Handbook/blob/main/llm_engineering/settings.py\"\u003esettings.py\u003c/a\u003e. Any variable in the \u003ccode\u003eSettings\u003c/code\u003e class can be configured through the \u003ccode\u003e.env\u003c/code\u003e file.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch2 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003e🏗️ Infrastructure\u003c/h2\u003e\u003ca id=\"user-content-️-infrastructure\" class=\"anchor\" aria-label=\"Permalink: 🏗️ Infrastructure\" href=\"#️-infrastructure\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch3 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eLocal infrastructure (for testing and development)\u003c/h3\u003e\u003ca id=\"user-content-local-infrastructure-for-testing-and-development\" class=\"anchor\" aria-label=\"Permalink: Local infrastructure (for testing and development)\" href=\"#local-infrastructure-for-testing-and-development\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eWhen running the project locally, we host a MongoDB and Qdrant database using Docker. Also, a testing ZenML server is made available through their Python package.\u003c/p\u003e\n\u003cdiv class=\"markdown-alert markdown-alert-warning\" dir=\"auto\"\u003e\u003cp class=\"markdown-alert-title\" dir=\"auto\"\u003e\u003csvg class=\"octicon octicon-alert mr-2\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"M6.457 1.047c.659-1.234 2.427-1.234 3.086 0l6.082 11.378A1.75 1.75 0 0 1 14.082 15H1.918a1.75 1.75 0 0 1-1.543-2.575Zm1.763.707a.25.25 0 0 0-.44 0L1.698 13.132a.25.25 0 0 0 .22.368h12.164a.25.25 0 0 0 .22-.368Zm.53 3.996v2.5a.75.75 0 0 1-1.5 0v-2.5a.75.75 0 0 1 1.5 0ZM9 11a1 1 0 1 1-2 0 1 1 0 0 1 2 0Z\"\u003e\u003c/path\u003e\u003c/svg\u003eWarning\u003c/p\u003e\u003cp dir=\"auto\"\u003eYou need Docker installed (\u0026gt;= v27.1.1)\u003c/p\u003e\n\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eFor ease of use, you can start the whole local development infrastructure with the following command:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe local-infrastructure-up\"\u003e\u003cpre\u003epoetry poe local-infrastructure-up\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eAlso, you can stop the ZenML server and all the Docker containers using the following command:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe local-infrastructure-down\"\u003e\u003cpre\u003epoetry poe local-infrastructure-down\u003c/pre\u003e\u003c/div\u003e\n\u003cdiv class=\"markdown-alert markdown-alert-warning\" dir=\"auto\"\u003e\u003cp class=\"markdown-alert-title\" dir=\"auto\"\u003e\u003csvg class=\"octicon octicon-alert mr-2\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"M6.457 1.047c.659-1.234 2.427-1.234 3.086 0l6.082 11.378A1.75 1.75 0 0 1 14.082 15H1.918a1.75 1.75 0 0 1-1.543-2.575Zm1.763.707a.25.25 0 0 0-.44 0L1.698 13.132a.25.25 0 0 0 .22.368h12.164a.25.25 0 0 0 .22-.368Zm.53 3.996v2.5a.75.75 0 0 1-1.5 0v-2.5a.75.75 0 0 1 1.5 0ZM9 11a1 1 0 1 1-2 0 1 1 0 0 1 2 0Z\"\u003e\u003c/path\u003e\u003c/svg\u003eWarning\u003c/p\u003e\u003cp dir=\"auto\"\u003eWhen running on MacOS, before starting the server, export the following environment variable:\n\u003ccode\u003eexport OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES\u003c/code\u003e\nOtherwise, the connection between the local server and pipeline will break. 🔗 More details in \u003ca href=\"https://github.com/zenml-io/zenml/issues/2369\" data-hovercard-type=\"issue\" data-hovercard-url=\"/zenml-io/zenml/issues/2369/hovercard\"\u003ethis issue\u003c/a\u003e.\nThis is done by default when using Poe the Poet.\u003c/p\u003e\n\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eStart the inference real-time RESTful API:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe run-inference-ml-service\"\u003e\u003cpre\u003epoetry poe run-inference-ml-service\u003c/pre\u003e\u003c/div\u003e\n\u003cdiv class=\"markdown-alert markdown-alert-important\" dir=\"auto\"\u003e\u003cp class=\"markdown-alert-title\" dir=\"auto\"\u003e\u003csvg class=\"octicon octicon-report mr-2\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"M0 1.75C0 .784.784 0 1.75 0h12.5C15.216 0 16 .784 16 1.75v9.5A1.75 1.75 0 0 1 14.25 13H8.06l-2.573 2.573A1.458 1.458 0 0 1 3 14.543V13H1.75A1.75 1.75 0 0 1 0 11.25Zm1.75-.25a.25.25 0 0 0-.25.25v9.5c0 .138.112.25.25.25h2a.75.75 0 0 1 .75.75v2.19l2.72-2.72a.749.749 0 0 1 .53-.22h6.5a.25.25 0 0 0 .25-.25v-9.5a.25.25 0 0 0-.25-.25Zm7 2.25v2.5a.75.75 0 0 1-1.5 0v-2.5a.75.75 0 0 1 1.5 0ZM9 9a1 1 0 1 1-2 0 1 1 0 0 1 2 0Z\"\u003e\u003c/path\u003e\u003c/svg\u003eImportant\u003c/p\u003e\u003cp dir=\"auto\"\u003eThe LLM microservice, called by the RESTful API, will work only after deploying the LLM to AWS SageMaker.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch4 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eZenML\u003c/h4\u003e\u003ca id=\"user-content-zenml\" class=\"anchor\" aria-label=\"Permalink: ZenML\" href=\"#zenml\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eDashboard URL: \u003ccode\u003elocalhost:8237\u003c/code\u003e\u003c/p\u003e\n\u003cp dir=\"auto\"\u003eDefault credentials:\u003c/p\u003e\n\u003cul dir=\"auto\"\u003e\n\u003cli\u003e\u003ccode\u003eusername\u003c/code\u003e: default\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003epassword\u003c/code\u003e:\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp dir=\"auto\"\u003e→ Find out more about using and setting up \u003ca href=\"https://docs.zenml.io/\" rel=\"nofollow\"\u003eZenML\u003c/a\u003e.\u003c/p\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch4 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eQdrant\u003c/h4\u003e\u003ca id=\"user-content-qdrant-1\" class=\"anchor\" aria-label=\"Permalink: Qdrant\" href=\"#qdrant-1\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eREST API URL: \u003ccode\u003elocalhost:6333\u003c/code\u003e\u003c/p\u003e\n\u003cp dir=\"auto\"\u003eDashboard URL: \u003ccode\u003elocalhost:6333/dashboard\u003c/code\u003e\u003c/p\u003e\n\u003cp dir=\"auto\"\u003e→ Find out more about using and setting up \u003ca href=\"https://qdrant.tech/documentation/quick-start/\" rel=\"nofollow\"\u003eQdrant with Docker\u003c/a\u003e.\u003c/p\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch4 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eMongoDB\u003c/h4\u003e\u003ca id=\"user-content-mongodb-1\" class=\"anchor\" aria-label=\"Permalink: MongoDB\" href=\"#mongodb-1\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eDatabase URI: \u003ccode\u003emongodb://llm_engineering:[email protected]:27017\u003c/code\u003e\u003c/p\u003e\n\u003cp dir=\"auto\"\u003eDatabase name: \u003ccode\u003etwin\u003c/code\u003e\u003c/p\u003e\n\u003cp dir=\"auto\"\u003eDefault credentials:\u003c/p\u003e\n\u003cul dir=\"auto\"\u003e\n\u003cli\u003e\u003ccode\u003eusername\u003c/code\u003e: llm_engineering\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003epassword\u003c/code\u003e: llm_engineering\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp dir=\"auto\"\u003e→ Find out more about using and setting up \u003ca href=\"https://www.mongodb.com/docs/manual/tutorial/install-mongodb-community-with-docker\" rel=\"nofollow\"\u003eMongoDB with Docker\u003c/a\u003e.\u003c/p\u003e\n\u003cp dir=\"auto\"\u003eYou can search your MongoDB collections using your \u003cstrong\u003eIDEs MongoDB plugin\u003c/strong\u003e (which you have to install separately), where you have to use the database URI to connect to the MongoDB database hosted within the Docker container: \u003ccode\u003emongodb://llm_engineering:[email protected]:27017\u003c/code\u003e\u003c/p\u003e\n\u003cdiv class=\"markdown-alert markdown-alert-important\" dir=\"auto\"\u003e\u003cp class=\"markdown-alert-title\" dir=\"auto\"\u003e\u003csvg class=\"octicon octicon-report mr-2\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"M0 1.75C0 .784.784 0 1.75 0h12.5C15.216 0 16 .784 16 1.75v9.5A1.75 1.75 0 0 1 14.25 13H8.06l-2.573 2.573A1.458 1.458 0 0 1 3 14.543V13H1.75A1.75 1.75 0 0 1 0 11.25Zm1.75-.25a.25.25 0 0 0-.25.25v9.5c0 .138.112.25.25.25h2a.75.75 0 0 1 .75.75v2.19l2.72-2.72a.749.749 0 0 1 .53-.22h6.5a.25.25 0 0 0 .25-.25v-9.5a.25.25 0 0 0-.25-.25Zm7 2.25v2.5a.75.75 0 0 1-1.5 0v-2.5a.75.75 0 0 1 1.5 0ZM9 9a1 1 0 1 1-2 0 1 1 0 0 1 2 0Z\"\u003e\u003c/path\u003e\u003c/svg\u003eImportant\u003c/p\u003e\u003cp dir=\"auto\"\u003eEverything related to training or running the LLMs (e.g., training, evaluation, inference) can only be run if you set up AWS SageMaker, as explained in the next section on cloud infrastructure.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch3 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eCloud infrastructure (for production)\u003c/h3\u003e\u003ca id=\"user-content-cloud-infrastructure-for-production\" class=\"anchor\" aria-label=\"Permalink: Cloud infrastructure (for production)\" href=\"#cloud-infrastructure-for-production\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eHere we will quickly present how to deploy the project to AWS and other serverless services. We won't go into the details (as everything is presented in the book) but only point out the main steps you have to go through.\u003c/p\u003e\n\u003cp dir=\"auto\"\u003eFirst, reinstall your Python dependencies with the AWS group:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry install --with aws\"\u003e\u003cpre\u003epoetry install --with aws\u003c/pre\u003e\u003c/div\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch4 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eAWS SageMaker\u003c/h4\u003e\u003ca id=\"user-content-aws-sagemaker\" class=\"anchor\" aria-label=\"Permalink: AWS SageMaker\" href=\"#aws-sagemaker\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cdiv class=\"markdown-alert markdown-alert-note\" dir=\"auto\"\u003e\u003cp class=\"markdown-alert-title\" dir=\"auto\"\u003e\u003csvg class=\"octicon octicon-info mr-2\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"M0 8a8 8 0 1 1 16 0A8 8 0 0 1 0 8Zm8-6.5a6.5 6.5 0 1 0 0 13 6.5 6.5 0 0 0 0-13ZM6.5 7.75A.75.75 0 0 1 7.25 7h1a.75.75 0 0 1 .75.75v2.75h.25a.75.75 0 0 1 0 1.5h-2a.75.75 0 0 1 0-1.5h.25v-2h-.25a.75.75 0 0 1-.75-.75ZM8 6a1 1 0 1 1 0-2 1 1 0 0 1 0 2Z\"\u003e\u003c/path\u003e\u003c/svg\u003eNote\u003c/p\u003e\u003cp dir=\"auto\"\u003eChapter 10 provides step-by-step instructions in the section \"Implementing the LLM microservice using AWS SageMaker\".\u003c/p\u003e\n\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eBy this point, we expect you to have AWS CLI installed and your AWS CLI and project's env vars (within the \u003ccode\u003e.env\u003c/code\u003e file) properly configured with an AWS admin user.\u003c/p\u003e\n\u003cp dir=\"auto\"\u003eTo ensure best practices, we must create a new AWS user restricted to creating and deleting only resources related to AWS SageMaker. Create it by running:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe create-sagemaker-role\"\u003e\u003cpre\u003epoetry poe create-sagemaker-role\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eIt will create a \u003ccode\u003esagemaker_user_credentials.json\u003c/code\u003e file at the root of your repository with your new \u003ccode\u003eAWS_ACCESS_KEY\u003c/code\u003e and \u003ccode\u003eAWS_SECRET_KEY\u003c/code\u003e values. \u003cstrong\u003eBut before replacing your new AWS credentials, also run the following command to create the execution role (to create it using your admin credentials).\u003c/strong\u003e\u003c/p\u003e\n\u003cp dir=\"auto\"\u003eTo create the IAM execution role used by AWS SageMaker to access other AWS resources on our behalf, run the following:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe create-sagemaker-execution-role\"\u003e\u003cpre\u003epoetry poe create-sagemaker-execution-role\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eIt will create a \u003ccode\u003esagemaker_execution_role.json\u003c/code\u003e file at the root of your repository with your new \u003ccode\u003eAWS_ARN_ROLE\u003c/code\u003e value. Add it to your \u003ccode\u003e.env\u003c/code\u003e file.\u003c/p\u003e\n\u003cp dir=\"auto\"\u003eOnce you've updated the \u003ccode\u003eAWS_ACCESS_KEY\u003c/code\u003e, \u003ccode\u003eAWS_SECRET_KEY\u003c/code\u003e, and \u003ccode\u003eAWS_ARN_ROLE\u003c/code\u003e values in your \u003ccode\u003e.env\u003c/code\u003e file, you can use AWS SageMaker. \u003cstrong\u003eNote that this step is crucial to complete the AWS setup.\u003c/strong\u003e\u003c/p\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch4 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eTraining\u003c/h4\u003e\u003ca id=\"user-content-training\" class=\"anchor\" aria-label=\"Permalink: Training\" href=\"#training\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eWe start the training pipeline through ZenML by running the following:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe run-training-pipeline\"\u003e\u003cpre\u003epoetry poe run-training-pipeline\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eThis will start the training code using the configs from \u003ccode\u003econfigs/training.yaml\u003c/code\u003e directly in SageMaker. You can visualize the results in Comet ML's dashboard.\u003c/p\u003e\n\u003cp dir=\"auto\"\u003eWe start the evaluation pipeline through ZenML by running the following:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe run-evaluation-pipeline\"\u003e\u003cpre\u003epoetry poe run-evaluation-pipeline\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eThis will start the evaluation code using the configs from \u003ccode\u003econfigs/evaluating.yaml\u003c/code\u003e directly in SageMaker. You can visualize the results in \u003ccode\u003e*-results\u003c/code\u003e datasets saved to your Hugging Face profile.\u003c/p\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch4 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eInference\u003c/h4\u003e\u003ca id=\"user-content-inference\" class=\"anchor\" aria-label=\"Permalink: Inference\" href=\"#inference\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eTo create an AWS SageMaker Inference Endpoint, run:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe deploy-inference-endpoint\"\u003e\u003cpre\u003epoetry poe deploy-inference-endpoint\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eTo test it out, run:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe test-sagemaker-endpoint\"\u003e\u003cpre\u003epoetry poe test-sagemaker-endpoint\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eTo delete it, run:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe delete-inference-endpoint\"\u003e\u003cpre\u003epoetry poe delete-inference-endpoint\u003c/pre\u003e\u003c/div\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch4 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eAWS: ML pipelines, artifacts, and containers\u003c/h4\u003e\u003ca id=\"user-content-aws-ml-pipelines-artifacts-and-containers\" class=\"anchor\" aria-label=\"Permalink: AWS: ML pipelines, artifacts, and containers\" href=\"#aws-ml-pipelines-artifacts-and-containers\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eThe ML pipelines, artifacts, and containers are deployed to AWS by leveraging ZenML's deployment features. Thus, you must create an account with ZenML Cloud and follow their guide on deploying a ZenML stack to AWS. Otherwise, we provide step-by-step instructions in \u003cstrong\u003eChapter 11\u003c/strong\u003e, section \u003cstrong\u003eDeploying the LLM Twin's pipelines to the cloud\u003c/strong\u003e on what you must do.\u003c/p\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch4 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eQdrant \u0026amp; MongoDB\u003c/h4\u003e\u003ca id=\"user-content-qdrant--mongodb\" class=\"anchor\" aria-label=\"Permalink: Qdrant \u0026amp; MongoDB\" href=\"#qdrant--mongodb\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eWe leverage Qdrant's and MongoDB's serverless options when deploying the project. Thus, you can either follow \u003ca href=\"https://qdrant.tech/documentation/cloud/create-cluster/\" rel=\"nofollow\"\u003eQdrant's\u003c/a\u003e and \u003ca href=\"https://www.mongodb.com/resources/products/fundamentals/mongodb-cluster-setup\" rel=\"nofollow\"\u003eMongoDB's\u003c/a\u003e tutorials on how to create a freemium cluster for each or go through \u003cstrong\u003eChapter 11\u003c/strong\u003e, section \u003cstrong\u003eDeploying the LLM Twin's pipelines to the cloud\u003c/strong\u003e and follow our step-by-step instructions.\u003c/p\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch4 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eGitHub Actions\u003c/h4\u003e\u003ca id=\"user-content-github-actions\" class=\"anchor\" aria-label=\"Permalink: GitHub Actions\" href=\"#github-actions\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eWe use GitHub Actions to implement our CI/CD pipelines. To implement your own, you have to fork our repository and set the following env vars as Actions secrets in your forked repository:\u003c/p\u003e\n\u003cul dir=\"auto\"\u003e\n\u003cli\u003e\u003ccode\u003eAWS_ACCESS_KEY_ID\u003c/code\u003e\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eAWS_SECRET_ACCESS_KEY\u003c/code\u003e\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eAWS_ECR_NAME\u003c/code\u003e\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eAWS_REGION\u003c/code\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp dir=\"auto\"\u003eAlso, we provide instructions on how to set everything up in \u003cstrong\u003eChapter 11\u003c/strong\u003e, section \u003cstrong\u003eAdding LLMOps to the LLM Twin\u003c/strong\u003e.\u003c/p\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch4 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eComet ML \u0026amp; Opik\u003c/h4\u003e\u003ca id=\"user-content-comet-ml--opik-1\" class=\"anchor\" aria-label=\"Permalink: Comet ML \u0026amp; Opik\" href=\"#comet-ml--opik-1\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eYou can visualize the results on their self-hosted dashboards if you create a Comet account and correctly set the \u003ccode\u003eCOMET_API_KEY\u003c/code\u003e env var. As Opik is powered by Comet, you don't have to set up anything else along Comet:\u003c/p\u003e\n\u003cul dir=\"auto\"\u003e\n\u003cli\u003e\u003ca href=\"https://www.comet.com/?utm_source=llm_handbook\u0026amp;utm_medium=github\u0026amp;utm_campaign=opik\" rel=\"nofollow\"\u003eComet ML (for experiment tracking)\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://www.comet.com/opik?utm_source=llm_handbook\u0026amp;utm_medium=github\u0026amp;utm_campaign=opik\" rel=\"nofollow\"\u003eOpik (for prompt monitoring)\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch3 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003e💰 Running the Project Costs\u003c/h3\u003e\u003ca id=\"user-content--running-the-project-costs\" class=\"anchor\" aria-label=\"Permalink: 💰 Running the Project Costs\" href=\"#-running-the-project-costs\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eWe will mostly stick to free tiers for all the services except for AWS and OpenAI's API, which are both pay-as-you-go services. The cost of running the project once, with our default values, will be roughly ~$25 (most of it comes from using AWS SageMaker for training and inference).\u003c/p\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch2 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003e⚡ Pipelines\u003c/h2\u003e\u003ca id=\"user-content--pipelines\" class=\"anchor\" aria-label=\"Permalink: ⚡ Pipelines\" href=\"#-pipelines\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eAll the ML pipelines will be orchestrated behind the scenes by \u003ca href=\"https://www.zenml.io/\" rel=\"nofollow\"\u003eZenML\u003c/a\u003e. A few exceptions exist when running utility scrips, such as exporting or importing from the data warehouse.\u003c/p\u003e\n\u003cp dir=\"auto\"\u003eThe ZenML pipelines are the entry point for most processes throughout this project. They are under the \u003ccode\u003epipelines/\u003c/code\u003e folder. Thus, when you want to understand or debug a workflow, starting with the ZenML pipeline is the best approach.\u003c/p\u003e\n\u003cp dir=\"auto\"\u003eTo see the pipelines running and their results:\u003c/p\u003e\n\u003cul dir=\"auto\"\u003e\n\u003cli\u003ego to your ZenML dashboard\u003c/li\u003e\n\u003cli\u003ego to the \u003ccode\u003ePipelines\u003c/code\u003e section\u003c/li\u003e\n\u003cli\u003eclick on a specific pipeline (e.g., \u003ccode\u003efeature_engineering\u003c/code\u003e)\u003c/li\u003e\n\u003cli\u003eclick on a specific run (e.g., \u003ccode\u003efeature_engineering_run_2024_06_20_18_40_24\u003c/code\u003e)\u003c/li\u003e\n\u003cli\u003eclick on a specific step or artifact of the DAG to find more details about it\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp dir=\"auto\"\u003eNow, let's explore all the pipelines you can run. From data collection to training, we will present them in their natural order to go through the LLM project end-to-end.\u003c/p\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch3 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eData pipelines\u003c/h3\u003e\u003ca id=\"user-content-data-pipelines\" class=\"anchor\" aria-label=\"Permalink: Data pipelines\" href=\"#data-pipelines\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eRun the data collection ETL:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe run-digital-data-etl\"\u003e\u003cpre\u003epoetry poe run-digital-data-etl\u003c/pre\u003e\u003c/div\u003e\n\u003cdiv class=\"markdown-alert markdown-alert-warning\" dir=\"auto\"\u003e\u003cp class=\"markdown-alert-title\" dir=\"auto\"\u003e\u003csvg class=\"octicon octicon-alert mr-2\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"M6.457 1.047c.659-1.234 2.427-1.234 3.086 0l6.082 11.378A1.75 1.75 0 0 1 14.082 15H1.918a1.75 1.75 0 0 1-1.543-2.575Zm1.763.707a.25.25 0 0 0-.44 0L1.698 13.132a.25.25 0 0 0 .22.368h12.164a.25.25 0 0 0 .22-.368Zm.53 3.996v2.5a.75.75 0 0 1-1.5 0v-2.5a.75.75 0 0 1 1.5 0ZM9 11a1 1 0 1 1-2 0 1 1 0 0 1 2 0Z\"\u003e\u003c/path\u003e\u003c/svg\u003eWarning\u003c/p\u003e\u003cp dir=\"auto\"\u003eYou must have Chrome (or another Chromium-based browser) installed on your system for LinkedIn and Medium crawlers to work (which use Selenium under the hood). Based on your Chrome version, the Chromedriver will be automatically installed to enable Selenium support. Another option is to run everything using our Docker image if you don't want to install Chrome. For example, to run all the pipelines combined you can run \u003ccode\u003epoetry poe run-docker-end-to-end-data-pipeline\u003c/code\u003e. Note that the command can be tweaked to support any other pipeline.\u003c/p\u003e\n\u003cp dir=\"auto\"\u003eIf, for any other reason, you don't have a Chromium-based browser installed and don't want to use Docker, you have two other options to bypass this Selenium issue:\u003c/p\u003e\n\u003cul dir=\"auto\"\u003e\n\u003cli\u003eComment out all the code related to Selenium, Chrome and all the links that use Selenium to crawl them (e.g., Medium), such as the \u003ccode\u003echromedriver_autoinstaller.install()\u003c/code\u003e command from \u003ca href=\"https://github.com/PacktPublishing/LLM-Engineers-Handbook/blob/main/llm_engineering/application/crawlers/base.py\"\u003eapplication.crawlers.base\u003c/a\u003e and other static calls that check for Chrome drivers and Selenium.\u003c/li\u003e\n\u003cli\u003eInstall Google Chrome using your CLI in environments such as GitHub Codespaces or other cloud VMs using the same command as in our \u003ca href=\"https://github.com/PacktPublishing/LLM-Engineers-Handbook/blob/main/Dockerfile#L10\"\u003eDocker file\u003c/a\u003e.\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eTo add additional links to collect from, go to \u003ccode\u003econfigs/digital_data_etl_[author_name].yaml\u003c/code\u003e and add them to the \u003ccode\u003elinks\u003c/code\u003e field. Also, you can create a completely new file and specify it at run time, like this: \u003ccode\u003epython -m llm_engineering.interfaces.orchestrator.run --run-etl --etl-config-filename configs/digital_data_etl_[your_name].yaml\u003c/code\u003e\u003c/p\u003e\n\u003cp dir=\"auto\"\u003eRun the feature engineering pipeline:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe run-feature-engineering-pipeline\"\u003e\u003cpre\u003epoetry poe run-feature-engineering-pipeline\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eGenerate the instruct dataset:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe run-generate-instruct-datasets-pipeline\"\u003e\u003cpre\u003epoetry poe run-generate-instruct-datasets-pipeline\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eGenerate the preference dataset:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe run-generate-preference-datasets-pipeline\"\u003e\u003cpre\u003epoetry poe run-generate-preference-datasets-pipeline\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eRun all of the above compressed into a single pipeline:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe run-end-to-end-data-pipeline\"\u003e\u003cpre\u003epoetry poe run-end-to-end-data-pipeline\u003c/pre\u003e\u003c/div\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch3 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eUtility pipelines\u003c/h3\u003e\u003ca id=\"user-content-utility-pipelines\" class=\"anchor\" aria-label=\"Permalink: Utility pipelines\" href=\"#utility-pipelines\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eExport the data from the data warehouse to JSON files:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe run-export-data-warehouse-to-json\"\u003e\u003cpre\u003epoetry poe run-export-data-warehouse-to-json\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eImport data to the data warehouse from JSON files (by default, it imports the data from the \u003ccode\u003edata/data_warehouse_raw_data\u003c/code\u003e directory):\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe run-import-data-warehouse-from-json\"\u003e\u003cpre\u003epoetry poe run-import-data-warehouse-from-json\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eExport ZenML artifacts to JSON:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe run-export-artifact-to-json-pipeline\"\u003e\u003cpre\u003epoetry poe run-export-artifact-to-json-pipeline\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eThis will export the following ZenML artifacts to the \u003ccode\u003eoutput\u003c/code\u003e folder as JSON files (it will take their latest version):\u003c/p\u003e\n\u003cul dir=\"auto\"\u003e\n\u003cli\u003ecleaned_documents.json\u003c/li\u003e\n\u003cli\u003einstruct_datasets.json\u003c/li\u003e\n\u003cli\u003epreference_datasets.json\u003c/li\u003e\n\u003cli\u003eraw_documents.json\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp dir=\"auto\"\u003eYou can configure what artifacts to export by tweaking the \u003ccode\u003econfigs/export_artifact_to_json.yaml\u003c/code\u003e configuration file.\u003c/p\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch3 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eTraining pipelines\u003c/h3\u003e\u003ca id=\"user-content-training-pipelines\" class=\"anchor\" aria-label=\"Permalink: Training pipelines\" href=\"#training-pipelines\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eRun the training pipeline:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe run-training-pipeline\"\u003e\u003cpre\u003epoetry poe run-training-pipeline\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eRun the evaluation pipeline:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe run-evaluation-pipeline\"\u003e\u003cpre\u003epoetry poe run-evaluation-pipeline\u003c/pre\u003e\u003c/div\u003e\n\u003cdiv class=\"markdown-alert markdown-alert-warning\" dir=\"auto\"\u003e\u003cp class=\"markdown-alert-title\" dir=\"auto\"\u003e\u003csvg class=\"octicon octicon-alert mr-2\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"M6.457 1.047c.659-1.234 2.427-1.234 3.086 0l6.082 11.378A1.75 1.75 0 0 1 14.082 15H1.918a1.75 1.75 0 0 1-1.543-2.575Zm1.763.707a.25.25 0 0 0-.44 0L1.698 13.132a.25.25 0 0 0 .22.368h12.164a.25.25 0 0 0 .22-.368Zm.53 3.996v2.5a.75.75 0 0 1-1.5 0v-2.5a.75.75 0 0 1 1.5 0ZM9 11a1 1 0 1 1-2 0 1 1 0 0 1 2 0Z\"\u003e\u003c/path\u003e\u003c/svg\u003eWarning\u003c/p\u003e\u003cp dir=\"auto\"\u003eFor this to work, make sure you properly configured AWS SageMaker as described in \u003ca href=\"#set-up-cloud-infrastructure-for-production\"\u003eSet up cloud infrastructure (for production)\u003c/a\u003e.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch3 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eInference pipelines\u003c/h3\u003e\u003ca id=\"user-content-inference-pipelines\" class=\"anchor\" aria-label=\"Permalink: Inference pipelines\" href=\"#inference-pipelines\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eCall the RAG retrieval module with a test query:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe call-rag-retrieval-module\"\u003e\u003cpre\u003epoetry poe call-rag-retrieval-module\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eStart the inference real-time RESTful API:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe run-inference-ml-service\"\u003e\u003cpre\u003epoetry poe run-inference-ml-service\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eCall the inference real-time RESTful API with a test query:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe call-inference-ml-service\"\u003e\u003cpre\u003epoetry poe call-inference-ml-service\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eRemember that you can monitor the prompt traces on \u003ca href=\"https://www.comet.com/opik\" rel=\"nofollow\"\u003eOpik\u003c/a\u003e.\u003c/p\u003e\n\u003cdiv class=\"markdown-alert markdown-alert-warning\" dir=\"auto\"\u003e\u003cp class=\"markdown-alert-title\" dir=\"auto\"\u003e\u003csvg class=\"octicon octicon-alert mr-2\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"M6.457 1.047c.659-1.234 2.427-1.234 3.086 0l6.082 11.378A1.75 1.75 0 0 1 14.082 15H1.918a1.75 1.75 0 0 1-1.543-2.575Zm1.763.707a.25.25 0 0 0-.44 0L1.698 13.132a.25.25 0 0 0 .22.368h12.164a.25.25 0 0 0 .22-.368Zm.53 3.996v2.5a.75.75 0 0 1-1.5 0v-2.5a.75.75 0 0 1 1.5 0ZM9 11a1 1 0 1 1-2 0 1 1 0 0 1 2 0Z\"\u003e\u003c/path\u003e\u003c/svg\u003eWarning\u003c/p\u003e\u003cp dir=\"auto\"\u003eFor the inference service to work, you must have the LLM microservice deployed to AWS SageMaker, as explained in the setup cloud infrastructure section.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch3 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eLinting \u0026amp; formatting (QA)\u003c/h3\u003e\u003ca id=\"user-content-linting--formatting-qa\" class=\"anchor\" aria-label=\"Permalink: Linting \u0026amp; formatting (QA)\" href=\"#linting--formatting-qa\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eCheck or fix your linting issues:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe lint-check\npoetry poe lint-fix\"\u003e\u003cpre\u003epoetry poe lint-check\npoetry poe lint-fix\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eCheck or fix your formatting issues:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe format-check\npoetry poe format-fix\"\u003e\u003cpre\u003epoetry poe format-check\npoetry poe format-fix\u003c/pre\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eCheck the code for leaked credentials:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe gitleaks-check\"\u003e\u003cpre\u003epoetry poe gitleaks-check\u003c/pre\u003e\u003c/div\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch3 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eTests\u003c/h3\u003e\u003ca id=\"user-content-tests\" class=\"anchor\" aria-label=\"Permalink: Tests\" href=\"#tests\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eRun all the tests using the following command:\u003c/p\u003e\n\u003cdiv class=\"highlight highlight-source-shell notranslate position-relative overflow-auto\" dir=\"auto\" data-snippet-clipboard-copy-content=\"poetry poe test\"\u003e\u003cpre\u003epoetry poe \u003cspan class=\"pl-c1\"\u003etest\u003c/span\u003e\u003c/pre\u003e\u003c/div\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch2 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003e🏃 Run project\u003c/h2\u003e\u003ca id=\"user-content--run-project\" class=\"anchor\" aria-label=\"Permalink: 🏃 Run project\" href=\"#-run-project\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eBased on the setup and usage steps described above, assuming the local and cloud infrastructure works and the \u003ccode\u003e.env\u003c/code\u003e is filled as expected, follow the next steps to run the LLM system end-to-end:\u003c/p\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch3 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eData\u003c/h3\u003e\u003ca id=\"user-content-data\" class=\"anchor\" aria-label=\"Permalink: Data\" href=\"#data\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003col dir=\"auto\"\u003e\n\u003cli\u003e\n\u003cp dir=\"auto\"\u003eCollect data: \u003ccode\u003epoetry poe run-digital-data-etl\u003c/code\u003e\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp dir=\"auto\"\u003eCompute features: \u003ccode\u003epoetry poe run-feature-engineering-pipeline\u003c/code\u003e\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp dir=\"auto\"\u003eCompute instruct dataset: \u003ccode\u003epoetry poe run-generate-instruct-datasets-pipeline\u003c/code\u003e\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp dir=\"auto\"\u003eCompute preference alignment dataset: \u003ccode\u003epoetry poe run-generate-preference-datasets-pipeline\u003c/code\u003e\u003c/p\u003e\n\u003c/li\u003e\n\u003c/ol\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch3 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eTraining\u003c/h3\u003e\u003ca id=\"user-content-training-1\" class=\"anchor\" aria-label=\"Permalink: Training\" href=\"#training-1\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cdiv class=\"markdown-alert markdown-alert-important\" dir=\"auto\"\u003e\u003cp class=\"markdown-alert-title\" dir=\"auto\"\u003e\u003csvg class=\"octicon octicon-report mr-2\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"M0 1.75C0 .784.784 0 1.75 0h12.5C15.216 0 16 .784 16 1.75v9.5A1.75 1.75 0 0 1 14.25 13H8.06l-2.573 2.573A1.458 1.458 0 0 1 3 14.543V13H1.75A1.75 1.75 0 0 1 0 11.25Zm1.75-.25a.25.25 0 0 0-.25.25v9.5c0 .138.112.25.25.25h2a.75.75 0 0 1 .75.75v2.19l2.72-2.72a.749.749 0 0 1 .53-.22h6.5a.25.25 0 0 0 .25-.25v-9.5a.25.25 0 0 0-.25-.25Zm7 2.25v2.5a.75.75 0 0 1-1.5 0v-2.5a.75.75 0 0 1 1.5 0ZM9 9a1 1 0 1 1-2 0 1 1 0 0 1 2 0Z\"\u003e\u003c/path\u003e\u003c/svg\u003eImportant\u003c/p\u003e\u003cp dir=\"auto\"\u003eFrom now on, for these steps to work, you need to properly set up AWS SageMaker, such as running \u003ccode\u003epoetry install --with aws\u003c/code\u003e and filling in the AWS-related environment variables and configs.\u003c/p\u003e\n\u003c/div\u003e\n\u003col start=\"5\" dir=\"auto\"\u003e\n\u003cli\u003e\n\u003cp dir=\"auto\"\u003eSFT fine-tuning Llamma 3.1: \u003ccode\u003epoetry poe run-training-pipeline\u003c/code\u003e\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp dir=\"auto\"\u003eFor DPO, go to \u003ccode\u003econfigs/training.yaml\u003c/code\u003e, change \u003ccode\u003efinetuning_type\u003c/code\u003e to \u003ccode\u003edpo\u003c/code\u003e, and run \u003ccode\u003epoetry poe run-training-pipeline\u003c/code\u003e again\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp dir=\"auto\"\u003eEvaluate fine-tuned models: \u003ccode\u003epoetry poe run-evaluation-pipeline\u003c/code\u003e\u003c/p\u003e\n\u003c/li\u003e\n\u003c/ol\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch3 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003eInference\u003c/h3\u003e\u003ca id=\"user-content-inference-1\" class=\"anchor\" aria-label=\"Permalink: Inference\" href=\"#inference-1\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cdiv class=\"markdown-alert markdown-alert-important\" dir=\"auto\"\u003e\u003cp class=\"markdown-alert-title\" dir=\"auto\"\u003e\u003csvg class=\"octicon octicon-report mr-2\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"M0 1.75C0 .784.784 0 1.75 0h12.5C15.216 0 16 .784 16 1.75v9.5A1.75 1.75 0 0 1 14.25 13H8.06l-2.573 2.573A1.458 1.458 0 0 1 3 14.543V13H1.75A1.75 1.75 0 0 1 0 11.25Zm1.75-.25a.25.25 0 0 0-.25.25v9.5c0 .138.112.25.25.25h2a.75.75 0 0 1 .75.75v2.19l2.72-2.72a.749.749 0 0 1 .53-.22h6.5a.25.25 0 0 0 .25-.25v-9.5a.25.25 0 0 0-.25-.25Zm7 2.25v2.5a.75.75 0 0 1-1.5 0v-2.5a.75.75 0 0 1 1.5 0ZM9 9a1 1 0 1 1-2 0 1 1 0 0 1 2 0Z\"\u003e\u003c/path\u003e\u003c/svg\u003eImportant\u003c/p\u003e\u003cp dir=\"auto\"\u003eFrom now on, for these steps to work, you need to properly set up AWS SageMaker, such as running \u003ccode\u003epoetry install --with aws\u003c/code\u003e and filling in the AWS-related environment variables and configs.\u003c/p\u003e\n\u003c/div\u003e\n\u003col start=\"8\" dir=\"auto\"\u003e\n\u003cli\u003e\n\u003cp dir=\"auto\"\u003eCall only the RAG retrieval module: \u003ccode\u003epoetry poe call-rag-retrieval-module\u003c/code\u003e\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp dir=\"auto\"\u003eDeploy the LLM Twin microservice to SageMaker: \u003ccode\u003epoetry poe deploy-inference-endpoint\u003c/code\u003e\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp dir=\"auto\"\u003eTest the LLM Twin microservice: \u003ccode\u003epoetry poe test-sagemaker-endpoint\u003c/code\u003e\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp dir=\"auto\"\u003eStart end-to-end RAG server: \u003ccode\u003epoetry poe run-inference-ml-service\u003c/code\u003e\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp dir=\"auto\"\u003eTest RAG server: \u003ccode\u003epoetry poe call-inference-ml-service\u003c/code\u003e\u003c/p\u003e\n\u003c/li\u003e\n\u003c/ol\u003e\n\u003cdiv class=\"markdown-heading\" dir=\"auto\"\u003e\u003ch2 tabindex=\"-1\" class=\"heading-element\" dir=\"auto\"\u003e📄 License\u003c/h2\u003e\u003ca id=\"user-content--license\" class=\"anchor\" aria-label=\"Permalink: 📄 License\" href=\"#-license\"\u003e\u003csvg class=\"octicon octicon-link\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" height=\"16\" aria-hidden=\"true\"\u003e\u003cpath d=\"m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z\"\u003e\u003c/path\u003e\u003c/svg\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cp dir=\"auto\"\u003eThis course is an open-source project released under the MIT license. Thus, as long you distribute our LICENSE and acknowledge our work, you can safely clone or fork this project and use it as a source of inspiration for whatever you want (e.g., university projects, college degree projects, personal projects, etc.).\u003c/p\u003e\n\u003c/article\u003e","loaded":true,"timedOut":false,"errorMessage":null,"headerInfo":{"toc":[{"level":1,"text":"👷 LLM Engineer's Handbook","anchor":"-llm-engineers-handbook","htmlText":"👷 LLM Engineer's Handbook"},{"level":2,"text":"🌟 Features","anchor":"-features","htmlText":"🌟 Features"},{"level":2,"text":"🔗 Dependencies","anchor":"-dependencies","htmlText":"🔗 Dependencies"},{"level":3,"text":"Local dependencies","anchor":"local-dependencies","htmlText":"Local dependencies"},{"level":3,"text":"Cloud services","anchor":"cloud-services","htmlText":"Cloud services"},{"level":2,"text":"🗂️ Project Structure","anchor":"️-project-structure","htmlText":"🗂️ Project Structure"},{"level":2,"text":"💻 Installation","anchor":"-installation","htmlText":"💻 Installation"},{"level":3,"text":"1. Clone the Repository","anchor":"1-clone-the-repository","htmlText":"1. Clone the Repository"},{"level":3,"text":"2. Set Up Python Environment","anchor":"2-set-up-python-environment","htmlText":"2. Set Up Python Environment"},{"level":4,"text":"Option A: Using Global Python (if version 3.11 is installed)","anchor":"option-a-using-global-python-if-version-311-is-installed","htmlText":"Option A: Using Global Python (if version 3.11 is installed)"},{"level":4,"text":"Option B: Using pyenv (recommended)","anchor":"option-b-using-pyenv-recommended","htmlText":"Option B: Using pyenv (recommended)"},{"level":3,"text":"3. Install Dependencies","anchor":"3-install-dependencies","htmlText":"3. Install Dependencies"},{"level":3,"text":"4. Activate the Environment","anchor":"4-activate-the-environment","htmlText":"4. Activate the Environment"},{"level":3,"text":"Alternative Command Execution","anchor":"alternative-command-execution","htmlText":"Alternative Command Execution"},{"level":4,"text":"Example:","anchor":"example","htmlText":"Example:"},{"level":3,"text":"5. Local Development Setup","anchor":"5-local-development-setup","htmlText":"5. Local Development Setup"},{"level":4,"text":"OpenAI","anchor":"openai","htmlText":"OpenAI"},{"level":4,"text":"Hugging Face","anchor":"hugging-face","htmlText":"Hugging Face"},{"level":4,"text":"Comet ML \u0026 Opik","anchor":"comet-ml--opik","htmlText":"Comet ML \u0026amp; Opik"},{"level":3,"text":"6. Deployment Setup","anchor":"6-deployment-setup","htmlText":"6. Deployment Setup"},{"level":4,"text":"MongoDB","anchor":"mongodb","htmlText":"MongoDB"},{"level":4,"text":"Qdrant","anchor":"qdrant","htmlText":"Qdrant"},{"level":4,"text":"AWS","anchor":"aws","htmlText":"AWS"},{"level":2,"text":"🏗️ Infrastructure","anchor":"️-infrastructure","htmlText":"🏗️ Infrastructure"},{"level":3,"text":"Local infrastructure (for testing and development)","anchor":"local-infrastructure-for-testing-and-development","htmlText":"Local infrastructure (for testing and development)"},{"level":4,"text":"ZenML","anchor":"zenml","htmlText":"ZenML"},{"level":4,"text":"Qdrant","anchor":"qdrant-1","htmlText":"Qdrant"},{"level":4,"text":"MongoDB","anchor":"mongodb-1","htmlText":"MongoDB"},{"level":3,"text":"Cloud infrastructure (for production)","anchor":"cloud-infrastructure-for-production","htmlText":"Cloud infrastructure (for production)"},{"level":4,"text":"AWS SageMaker","anchor":"aws-sagemaker","htmlText":"AWS SageMaker"},{"level":4,"text":"Training","anchor":"training","htmlText":"Training"},{"level":4,"text":"Inference","anchor":"inference","htmlText":"Inference"},{"level":4,"text":"AWS: ML pipelines, artifacts, and containers","anchor":"aws-ml-pipelines-artifacts-and-containers","htmlText":"AWS: ML pipelines, artifacts, and containers"},{"level":4,"text":"Qdrant \u0026 MongoDB","anchor":"qdrant--mongodb","htmlText":"Qdrant \u0026amp; MongoDB"},{"level":4,"text":"GitHub Actions","anchor":"github-actions","htmlText":"GitHub Actions"},{"level":4,"text":"Comet ML \u0026 Opik","anchor":"comet-ml--opik-1","htmlText":"Comet ML \u0026amp; Opik"},{"level":3,"text":"💰 Running the Project Costs","anchor":"-running-the-project-costs","htmlText":"💰 Running the Project Costs"},{"level":2,"text":"⚡ Pipelines","anchor":"-pipelines","htmlText":"⚡ Pipelines"},{"level":3,"text":"Data pipelines","anchor":"data-pipelines","htmlText":"Data pipelines"},{"level":3,"text":"Utility pipelines","anchor":"utility-pipelines","htmlText":"Utility pipelines"},{"level":3,"text":"Training pipelines","anchor":"training-pipelines","htmlText":"Training pipelines"},{"level":3,"text":"Inference pipelines","anchor":"inference-pipelines","htmlText":"Inference pipelines"},{"level":3,"text":"Linting \u0026 formatting (QA)","anchor":"linting--formatting-qa","htmlText":"Linting \u0026amp; formatting (QA)"},{"level":3,"text":"Tests","anchor":"tests","htmlText":"Tests"},{"level":2,"text":"🏃 Run project","anchor":"-run-project","htmlText":"🏃 Run project"},{"level":3,"text":"Data","anchor":"data","htmlText":"Data"},{"level":3,"text":"Training","anchor":"training-1","htmlText":"Training"},{"level":3,"text":"Inference","anchor":"inference-1","htmlText":"Inference"},{"level":2,"text":"📄 License","anchor":"-license","htmlText":"📄 License"}],"siteNavLoginPath":"/login?return_to=https%3A%2F%2Fgithub.com%2FPacktPublishing%2FLLM-Engineers-Handbook"}},{"displayName":"LICENSE","repoName":"LLM-Engineers-Handbook","refName":"main","path":"LICENSE","preferredFileType":"license","tabName":"MIT","richText":null,"loaded":false,"timedOut":false,"errorMessage":null,"headerInfo":{"toc":null,"siteNavLoginPath":"/login?return_to=https%3A%2F%2Fgithub.com%2FPacktPublishing%2FLLM-Engineers-Handbook"}}],"overviewFilesProcessingTime":0,"copilotSWEAgentEnabled":false}},"codeViewLayoutRoute":{"repo":{"id":784183059,"defaultBranch":"main","name":"LLM-Engineers-Handbook","ownerLogin":"PacktPublishing","currentUserCanPush":false,"isFork":false,"isEmpty":false,"createdAt":"2024-04-09T10:56:03.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/10974906?v=4","public":true,"private":false,"isOrgOwned":true},"currentUser":null,"uploadToken":"WOxTsk3ONpA8HId0LQWoFVtegXtQklU5KzEGakQRGMxXRLo7_A_KPj7Eimi2PcCycFiWK0I1c3cp84qYaQq9Tw","allShortcutsEnabled":false,"treeExpanded":true,"path":"/","symbolsExpanded":false,"refInfo":{"name":"main","listCacheKey":"v0:1735803499.0","canEdit":false,"currentOid":"e6038da300616020695f8effb3cbf0a78aac5870"},"helpUrl":"https://docs.github.com","findFileWorkerPath":"/assets-cdn/worker/find-file-worker-4e5d7136862a2a48.js","findInFileWorkerPath":"/assets-cdn/worker/find-in-file-worker-4c35b25d88167fef.js","githubDevUrl":null},"csrf_tokens":{"/PacktPublishing/LLM-Engineers-Handbook/branches":{"post":"7hAIiPebH9R5AgJNxe3XQbueE84wKOQ4HeVTs8e6prglNM2QtHf5NgDRXFX1F2ymEHBf4I-KhYXuoy-j_l_seg"}}},"title":"GitHub - PacktPublishing/LLM-Engineers-Handbook: The LLM's practical guide: From the fundamentals to deploying advanced LLM and RAG apps to AWS using LLMOps best practices","appPayload":{},"meta":{"title":"GitHub - PacktPublishing/LLM-Engineers-Handbook: The LLM's practical guide: From the fundamentals to deploying advanced LLM and RAG apps to AWS using LLMOps best practices"}} PacktPublishing/LLM-Engineers-Handbook mainBranchesTagsGo to fileCodeOpen more actions menuFolders and filesNameNameLast commit messageLast commit dateLatest commit History201 Commits201 Commits.github/workflows.github/workflows .vscode.vscode code_snippetscode_snippets configsconfigs datadata imagesimages llm_engineeringllm_engineering pipelinespipelines stepssteps teststests toolstools .env.example.env.example .gitignore.gitignore .pre-commit-config.yaml.pre-commit-config.yaml .python-version.python-version DockerfileDockerfile LICENSELICENSE README.mdREADME.md docker-compose.ymldocker-compose.yml poetry.lockpoetry.lock pyproject.tomlpyproject.toml ruff.tomlruff.toml View all filesRepository files navigationREADMEMIT license 👷 LLM Engineer's Handbook Official repository of the LLM Engineer's Handbook by Paul Iusztin and Maxime Labonne Find the book on Amazon or Packt 🌟 Features The goal of this book is to create your own end-to-end LLM-based system using best practices: 📝 Data collection & generation 🔄 LLM training pipeline 📊 Simple RAG system 🚀 Production-ready AWS deployment 🔍 Comprehensive monitoring 🧪 Testing and evaluation framework You can download and use the final trained model on Hugging Face. ImportantThe code in this GitHub repository is actively maintained and may contain updates not reflected in the book. Always refer to this repository for the latest version of the code. 🔗 Dependencies Local dependencies To install and run the project locally, you need the following dependencies. Tool Version Purpose Installation Link pyenv ≥2.3.36 Multiple Python versions (optional) Install Guide Python 3.11 Runtime environment Download Poetry >= 1.8.3 and < 2.0 Package management Install Guide Docker ≥27.1.1 Containerization Install Guide AWS CLI ≥2.15.42 Cloud management Install Guide Git ≥2.44.0 Version control Download Cloud services The code also uses and depends on the following cloud services. For now, you don't have to do anything. We will guide you in the installation and deployment sections on how to use them: Service Purpose HuggingFace Model registry Comet ML Experiment tracker Opik Prompt monitoring ZenML Orchestrator and artifacts layer AWS Compute and storage MongoDB NoSQL database Qdrant Vector database GitHub Actions CI/CD pipeline In the LLM Engineer's Handbook, Chapter 2 will walk you through each tool. Chapters 10 and 11 provide step-by-step guides on how to set up everything you need. 🗂️ Project Structure Here is the directory overview: . ├── code_snippets/ # Standalone example code ├── configs/ # Pipeline configuration files ├── llm_engineering/ # Core project package │ ├── application/ │ ├── domain/ │ ├── infrastructure/ │ ├── model/ ├── pipelines/ # ML pipeline definitions ├── steps/ # Pipeline components ├── tests/ # Test examples ├── tools/ # Utility scripts │ ├── run.py │ ├── ml_service.py │ ├── rag.py │ ├── data_warehouse.py llm_engineering/ is the main Python package implementing LLM and RAG functionality. It follows Domain-Driven Design (DDD) principles: domain/: Core business entities and structures application/: Business logic, crawlers, and RAG implementation model/: LLM training and inference infrastructure/: External service integrations (AWS, Qdrant, MongoDB, FastAPI) The code logic and imports flow as follows: infrastructure → model → application → domain pipelines/: Contains the ZenML ML pipelines, which serve as the entry point for all the ML pipelines. Coordinates the data processing and model training stages of the ML lifecycle. steps/: Contains individual ZenML steps, which are reusable components for building and customizing ZenML pipelines. Steps perform specific tasks (e.g., data loading, preprocessing) and can be combined within the ML pipelines. tests/: Covers a few sample tests used as examples within the CI pipeline. tools/: Utility scripts used to call the ZenML pipelines and inference code: run.py: Entry point script to run ZenML pipelines. ml_service.py: Starts the REST API inference server. rag.py: Demonstrates usage of the RAG retrieval module. data_warehouse.py: Used to export or import data from the MongoDB data warehouse through JSON files. configs/: ZenML YAML configuration files to control the execution of pipelines and steps. code_snippets/: Independent code examples that can be executed independently. 💻 Installation NoteIf you are experiencing issues while installing and running the repository, consider checking the Issues GitHub section for other people who solved similar problems or directly asking us for help. 1. Clone the Repository Start by cloning the repository and navigating to the project directory: git clone https://github.com/PacktPublishing/LLM-Engineers-Handbook.git cd LLM-Engineers-Handbook Next, we have to prepare your Python environment and its adjacent dependencies. 2. Set Up Python Environment The project requires Python 3.11. You can either use your global Python installation or set up a project-specific version using pyenv. Option A: Using Global Python (if version 3.11 is installed) Verify your Python version: python --version # Should show Python 3.11.x Option B: Using pyenv (recommended) Verify pyenv installation: pyenv --version # Should show pyenv 2.3.36 or later Install Python 3.11.8: pyenv install 3.11.8 Verify the installation: python --version # Should show Python 3.11.8 Confirm Python version in the project directory: python --version # Output: Python 3.11.8 NoteThe project includes a .python-version file that automatically sets the correct Python version when you're in the project directory. 3. Install Dependencies The project uses Poetry for dependency management. Verify Poetry installation: poetry --version # Should show Poetry version 1.8.3 or later Set up the project environment and install dependencies: poetry env use 3.11 poetry install --without aws poetry run pre-commit install This will: Configure Poetry to use Python 3.11 Install project dependencies (excluding AWS-specific packages) Set up pre-commit hooks for code verification 4. Activate the Environment As our task manager, we run all the scripts using Poe the Poet. Start a Poetry shell: poetry shell Run project commands using Poe the Poet: poetry poe ... 🔧 Troubleshooting Poe the Poet Installation Alternative Command Execution If you're experiencing issues with poethepoet, you can still run the project commands directly through Poetry. Here's how: Look up the command definition in pyproject.toml Use poetry run with the underlying command Example: Instead of: poetry poe local-infrastructure-up Use the direct command from pyproject.toml: poetry run <actual-command-from-pyproject-toml> Note: All project commands are defined in the [tool.poe.tasks] section of pyproject.toml Now, let's configure our local project with all the necessary credentials and tokens to run the code locally. 5. Local Development Setup After you have installed all the dependencies, you must create and fill a .env file with your credentials to appropriately interact with other services and run the project. Setting your sensitive credentials in a .env file is a good security practice, as this file won't be committed to GitHub or shared with anyone else. First, copy our example by running the following: cp .env.example .env # The file must be at your repository's root! Now, let's understand how to fill in all the essential variables within the .env file to get you started. The following are the mandatory settings we must complete when working locally: OpenAI To authenticate to OpenAI's API, you must fill out the OPENAI_API_KEY env var with an authentication token. OPENAI_API_KEY=your_api_key_here → Check out this tutorial to learn how to provide one from OpenAI. Hugging Face To authenticate to Hugging Face, you must fill out the HUGGINGFACE_ACCESS_TOKEN env var with an authentication token. HUGGINGFACE_ACCESS_TOKEN=your_token_here → Check out this tutorial to learn how to provide one from Hugging Face. Comet ML & Opik To authenticate to Comet ML (required only during training) and Opik, you must fill out the COMET_API_KEY env var with your authentication token. COMET_API_KEY=your_api_key_here → Check out this tutorial to learn how to get started with Opik. You can also access Opik's dashboard using 🔗this link. 6. Deployment Setup When deploying the project to the cloud, we must set additional settings for Mongo, Qdrant, and AWS. If you are just working locally, the default values of these env vars will work out of the box. Detailed deployment instructions are available in Chapter 11 of the LLM Engineer's Handbook. MongoDB We must change the DATABASE_HOST env var with the URL pointing to your cloud MongoDB cluster. DATABASE_HOST=your_mongodb_url → Check out this tutorial to learn how to create and host a MongoDB cluster for free. Qdrant Change USE_QDRANT_CLOUD to true, QDRANT_CLOUD_URL with the URL point to your cloud Qdrant cluster, and QDRANT_APIKEY with its API key. USE_QDRANT_CLOUD=true QDRANT_CLOUD_URL=your_qdrant_cloud_url QDRANT_APIKEY=your_qdrant_api_key → Check out this tutorial to learn how to create a Qdrant cluster for free AWS For your AWS set-up to work correctly, you need the AWS CLI installed on your local machine and properly configured with an admin user (or a user with enough permissions to create new SageMaker, ECR, and S3 resources; using an admin user will make everything more straightforward). Chapter 2 provides step-by-step instructions on how to install the AWS CLI, create an admin user on AWS, and get an access key to set up the AWS_ACCESS_KEY and AWS_SECRET_KEY environment variables. If you already have an AWS admin user in place, you have to configure the following env vars in your .env file: AWS_REGION=eu-central-1 # Change it with your AWS region. AWS_ACCESS_KEY=your_aws_access_key AWS_SECRET_KEY=your_aws_secret_key AWS credentials are typically stored in ~/.aws/credentials. You can view this file directly using cat or similar commands: cat ~/.aws/credentials ImportantAdditional configuration options are available in settings.py. Any variable in the Settings class can be configured through the .env file. 🏗️ Infrastructure Local infrastructure (for testing and development) When running the project locally, we host a MongoDB and Qdrant database using Docker. Also, a testing ZenML server is made available through their Python package. WarningYou need Docker installed (>= v27.1.1) For ease of use, you can start the whole local development infrastructure with the following command: poetry poe local-infrastructure-up Also, you can stop the ZenML server and all the Docker containers using the following command: poetry poe local-infrastructure-down WarningWhen running on MacOS, before starting the server, export the following environment variable: export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES Otherwise, the connection between the local server and pipeline will break. 🔗 More details in this issue. This is done by default when using Poe the Poet. Start the inference real-time RESTful API: poetry poe run-inference-ml-service ImportantThe LLM microservice, called by the RESTful API, will work only after deploying the LLM to AWS SageMaker. ZenML Dashboard URL: localhost:8237 Default credentials: username: default password: → Find out more about using and setting up ZenML. Qdrant REST API URL: localhost:6333 Dashboard URL: localhost:6333/dashboard → Find out more about using and setting up Qdrant with Docker. MongoDB Database URI: mongodb://llm_engineering:[email protected]:27017 Database name: twin Default credentials: username: llm_engineering password: llm_engineering → Find out more about using and setting up MongoDB with Docker. You can search your MongoDB collections using your IDEs MongoDB plugin (which you have to install separately), where you have to use the database URI to connect to the MongoDB database hosted within the Docker container: mongodb://llm_engineering:[email protected]:27017 ImportantEverything related to training or running the LLMs (e.g., training, evaluation, inference) can only be run if you set up AWS SageMaker, as explained in the next section on cloud infrastructure. Cloud infrastructure (for production) Here we will quickly present how to deploy the project to AWS and other serverless services. We won't go into the details (as everything is presented in the book) but only point out the main steps you have to go through. First, reinstall your Python dependencies with the AWS group: poetry install --with aws AWS SageMaker NoteChapter 10 provides step-by-step instructions in the section "Implementing the LLM microservice using AWS SageMaker". By this point, we expect you to have AWS CLI installed and your AWS CLI and project's env vars (within the .env file) properly configured with an AWS admin user. To ensure best practices, we must create a new AWS user restricted to creating and deleting only resources related to AWS SageMaker. Create it by running: poetry poe create-sagemaker-role It will create a sagemaker_user_credentials.json file at the root of your repository with your new AWS_ACCESS_KEY and AWS_SECRET_KEY values. But before replacing your new AWS credentials, also run the following command to create the execution role (to create it using your admin credentials). To create the IAM execution role used by AWS SageMaker to access other AWS resources on our behalf, run the following: poetry poe create-sagemaker-execution-role It will create a sagemaker_execution_role.json file at the root of your repository with your new AWS_ARN_ROLE value. Add it to your .env file. Once you've updated the AWS_ACCESS_KEY, AWS_SECRET_KEY, and AWS_ARN_ROLE values in your .env file, you can use AWS SageMaker. Note that this step is crucial to complete the AWS setup. Training We start the training pipeline through ZenML by running the following: poetry poe run-training-pipeline This will start the training code using the configs from configs/training.yaml directly in SageMaker. You can visualize the results in Comet ML's dashboard. We start the evaluation pipeline through ZenML by running the following: poetry poe run-evaluation-pipeline This will start the evaluation code using the configs from configs/evaluating.yaml directly in SageMaker. You can visualize the results in *-results datasets saved to your Hugging Face profile. Inference To create an AWS SageMaker Inference Endpoint, run: poetry poe deploy-inference-endpoint To test it out, run: poetry poe test-sagemaker-endpoint To delete it, run: poetry poe delete-inference-endpoint AWS: ML pipelines, artifacts, and containers The ML pipelines, artifacts, and containers are deployed to AWS by leveraging ZenML's deployment features. Thus, you must create an account with ZenML Cloud and follow their guide on deploying a ZenML stack to AWS. Otherwise, we provide step-by-step instructions in Chapter 11, section Deploying the LLM Twin's pipelines to the cloud on what you must do. Qdrant & MongoDB We leverage Qdrant's and MongoDB's serverless options when deploying the project. Thus, you can either follow Qdrant's and MongoDB's tutorials on how to create a freemium cluster for each or go through Chapter 11, section Deploying the LLM Twin's pipelines to the cloud and follow our step-by-step instructions. GitHub Actions We use GitHub Actions to implement our CI/CD pipelines. To implement your own, you have to fork our repository and set the following env vars as Actions secrets in your forked repository: AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_ECR_NAME AWS_REGION Also, we provide instructions on how to set everything up in Chapter 11, section Adding LLMOps to the LLM Twin. Comet ML & Opik You can visualize the results on their self-hosted dashboards if you create a Comet account and correctly set the COMET_API_KEY env var. As Opik is powered by Comet, you don't have to set up anything else along Comet: Comet ML (for experiment tracking) Opik (for prompt monitoring) 💰 Running the Project Costs We will mostly stick to free tiers for all the services except for AWS and OpenAI's API, which are both pay-as-you-go services. The cost of running the project once, with our default values, will be roughly ~$25 (most of it comes from using AWS SageMaker for training and inference). ⚡ Pipelines All the ML pipelines will be orchestrated behind the scenes by ZenML. A few exceptions exist when running utility scrips, such as exporting or importing from the data warehouse. The ZenML pipelines are the entry point for most processes throughout this project. They are under the pipelines/ folder. Thus, when you want to understand or debug a workflow, starting with the ZenML pipeline is the best approach. To see the pipelines running and their results: go to your ZenML dashboard go to the Pipelines section click on a specific pipeline (e.g., feature_engineering) click on a specific run (e.g., feature_engineering_run_2024_06_20_18_40_24) click on a specific step or artifact of the DAG to find more details about it Now, let's explore all the pipelines you can run. From data collection to training, we will present them in their natural order to go through the LLM project end-to-end. Data pipelines Run the data collection ETL: poetry poe run-digital-data-etl WarningYou must have Chrome (or another Chromium-based browser) installed on your system for LinkedIn and Medium crawlers to work (which use Selenium under the hood). Based on your Chrome version, the Chromedriver will be automatically installed to enable Selenium support. Another option is to run everything using our Docker image if you don't want to install Chrome. For example, to run all the pipelines combined you can run poetry poe run-docker-end-to-end-data-pipeline. Note that the command can be tweaked to support any other pipeline. If, for any other reason, you don't have a Chromium-based browser installed and don't want to use Docker, you have two other options to bypass this Selenium issue: Comment out all the code related to Selenium, Chrome and all the links that use Selenium to crawl them (e.g., Medium), such as the chromedriver_autoinstaller.install() command from application.crawlers.base and other static calls that check for Chrome drivers and Selenium. Install Google Chrome using your CLI in environments such as GitHub Codespaces or other cloud VMs using the same command as in our Docker file. To add additional links to collect from, go to configs/digital_data_etl_[author_name].yaml and add them to the links field. Also, you can create a completely new file and specify it at run time, like this: python -m llm_engineering.interfaces.orchestrator.run --run-etl --etl-config-filename configs/digital_data_etl_[your_name].yaml Run the feature engineering pipeline: poetry poe run-feature-engineering-pipeline Generate the instruct dataset: poetry poe run-generate-instruct-datasets-pipeline Generate the preference dataset: poetry poe run-generate-preference-datasets-pipeline Run all of the above compressed into a single pipeline: poetry poe run-end-to-end-data-pipeline Utility pipelines Export the data from the data warehouse to JSON files: poetry poe run-export-data-warehouse-to-json Import data to the data warehouse from JSON files (by default, it imports the data from the data/data_warehouse_raw_data directory): poetry poe run-import-data-warehouse-from-json Export ZenML artifacts to JSON: poetry poe run-export-artifact-to-json-pipeline This will export the following ZenML artifacts to the output folder as JSON files (it will take their latest version): cleaned_documents.json instruct_datasets.json preference_datasets.json raw_documents.json You can configure what artifacts to export by tweaking the configs/export_artifact_to_json.yaml configuration file. Training pipelines Run the training pipeline: poetry poe run-training-pipeline Run the evaluation pipeline: poetry poe run-evaluation-pipeline WarningFor this to work, make sure you properly configured AWS SageMaker as described in Set up cloud infrastructure (for production). Inference pipelines Call the RAG retrieval module with a test query: poetry poe call-rag-retrieval-module Start the inference real-time RESTful API: poetry poe run-inference-ml-service Call the inference real-time RESTful API with a test query: poetry poe call-inference-ml-service Remember that you can monitor the prompt traces on Opik. WarningFor the inference service to work, you must have the LLM microservice deployed to AWS SageMaker, as explained in the setup cloud infrastructure section. Linting & formatting (QA) Check or fix your linting issues: poetry poe lint-check poetry poe lint-fix Check or fix your formatting issues: poetry poe format-check poetry poe format-fix Check the code for leaked credentials: poetry poe gitleaks-check Tests Run all the tests using the following command: poetry poe test 🏃 Run project Based on the setup and usage steps described above, assuming the local and cloud infrastructure works and the .env is filled as expected, follow the next steps to run the LLM system end-to-end: Data Collect data: poetry poe run-digital-data-etl Compute features: poetry poe run-feature-engineering-pipeline Compute instruct dataset: poetry poe run-generate-instruct-datasets-pipeline Compute preference alignment dataset: poetry poe run-generate-preference-datasets-pipeline Training ImportantFrom now on, for these steps to work, you need to properly set up AWS SageMaker, such as running poetry install --with aws and filling in the AWS-related environment variables and configs. SFT fine-tuning Llamma 3.1: poetry poe run-training-pipeline For DPO, go to configs/training.yaml, change finetuning_type to dpo, and run poetry poe run-training-pipeline again Evaluate fine-tuned models: poetry poe run-evaluation-pipeline Inference ImportantFrom now on, for these steps to work, you need to properly set up AWS SageMaker, such as running poetry install --with aws and filling in the AWS-related environment variables and configs. Call only the RAG retrieval module: poetry poe call-rag-retrieval-module Deploy the LLM Twin microservice to SageMaker: poetry poe deploy-inference-endpoint Test the LLM Twin microservice: poetry poe test-sagemaker-endpoint Start end-to-end RAG server: poetry poe run-inference-ml-service Test RAG server: poetry poe call-inference-ml-service 📄 License This course is an open-source project released under the MIT license. Thus, as long you distribute our LICENSE and acknowledge our work, you can safely clone or fork this project and use it as a source of inspiration for whatever you want (e.g., university projects, college degree projects, personal projects, etc.). About The LLM's practical guide: From the fundamentals to deploying advanced LLM and RAG apps to AWS using LLMOps best practices www.amazon.com/LLM-Engineers-Handbook-engineering-production/dp/1836200072/ Topics aws rag mlops llm llmops genai fine-tuning-llm llm-evaluation ml-system-design Resources Readme License MIT license Uh oh! There was an error while loading. Please reload this page. Activity Custom properties Stars 4.9k stars Watchers 57 watching Forks 1.2k forks Report repository Releases No releases published Packages 0 Uh oh! There was an error while loading. Please reload this page. Uh oh! There was an error while loading. Please reload this page. Contributors Uh oh! There was an error while loading. Please reload this page. Languages Python 99.2% Dockerfile 0.8% {"resolvedServerColorMode":"day"} Footer © 2026 GitHub, Inc. Footer navigation Terms Privacy Security Status Community Docs Contact Manage cookies Do not share my personal information You can’t perform that action at this time.